In the “The tale of the deep learning model that failed my driving exam”, we compared the decisions of an 18-year-old driver and a deep learning (DL) model when facing a tricky intersection where the driver had a green light, but an ambulance unexpectedly crossed. While both drivers stopped at the intersection, when the driver evaluator asked the model to explain how it came to such a decision, he realized the model’s responses were purely technical, focused on pixel values and activations rather than a true understanding of traffic rules or emergency vehicles:

“If pixel 2890 had an RGB value of (28, 178, 111), the model would have chosen to cross.” [Example of a DL driver explanation]

To help the driving evaluator, in the last chapter, we introduced Concept Bottleneck Models (CBMs):

CBMs allow the evaluator to gain insights into a DL model’s decision-making process as:

• Task predictions can be traced back to the activation of human-interpretable concepts. This enables the model to answer the driving evaluator’s question by saying, “I decided to cross as I saw a green light and there was no ambulance”.
• Altering concept values changes the model’s decisions. This allows the model to respond to the evaluator’s question with, “In the same scenario, if there is an ambulance, I would cross”.

While these qualitative answers reassured the evaluator, he now needs a quantitative way to assess how well the model responds in different situations. To show this, we use three key metrics in concept-based interpretability: concept/task predictive performance, intervention effectiveness, and concept completeness.

In the following, we assume we have a CBM (θ₍g₎, θ₍f₎) and a dataset of i.i.d. triples (input, concepts, task) 𝒟 = {(𝒙, 𝒄, 𝓎)}, which we use to compute these metrics.

1. Predictive performance

Task performance

In the driving test, task performance [1] measures how well the model predicts whether to cross or stop at the intersection, based on concepts like traffic light color and the ambulance. A high task performance indicates that the model generally makes the correct decision. Task performance is usually represented as the likelihood:

Notice that in common classification settings, this likelihood becomes the empirical test task accuracy of the model.

Limitations: High task performance alone doesn’t guarantee interpretability. The model might achieve high performance by relying on uninterpretable or poorly defined concepts, potentially missing key information. For example, a CBM could correctly predict whether to cross or stop, but its decisions might be unaffected by changing the value of the concept “ambulance”.

Concept performance

Concept performance [1] measures how well the model correctly predicts the concepts from the input data. In our driving example, this evaluates how well the model detects traffic light color or the ambulance’s presence. Similarly to task performance, concept performance can be measured as the likelihood:

In practice, this likelihood may be replaced with proxy metrics such as mean Area Under the Receiving Operating Characteristic curve (AUC-ROC) across all concepts. This is particularly useful in situations where concepts are highly imbalanced, and therefore a majority classifier may achieve high likelihood without really learning anything.

Limitations: While high concept performance means the model can correctly identify the traffic light color and ambulance presence, it doesn’t always imply that the model will make the right final decision. For example, even if the model accurately identifies a green light and no ambulance, it might still incorrectly predict that the car should stop.

2. Intervention effectiveness

In the coding practice of the previous chapter, we observed how changing the predicted value of a concept affects task predictions. This property allows human experts to intervene on mispredicted concept values and correct them to improve the model’s task performance. Intervention effectiveness [1] measures how well a CBM improves task performance after such corrections. This can be achieved by computing the area under the intervention curve as follows:

• consider a subset of the powerset of the concept indices I ⊆ 𝒫({1, …, |𝒞|}) to intervene on;
• for each concept group I′ ∈ I, replace the subset of predictions g(𝒙)[I′] with ground truth values 𝒄[I′];

In the driving test scenario, imagine the model incorrectly predicts that the ambulance is absent, leading to a wrong decision to cross. This metric would evaluate whether correcting the concept by setting “ambulance = 1” leads the model to change its prediction to the correct one (i.e., stopping). After this human intervention, we expect the model to revise its decision and recommend stopping the car.

Intervention effectiveness is particularly useful when human experts need to intervene and adjust concept values to improve the model’s decisions, ensuring the model responds accurately after corrections.

Limitations: This metric is highly dependent on the quality of the interventions provided by humans. Poorly chosen interventions can reduce the model’s performance, even if the concepts themselves are correct. Moreover, if the concept predictive accuracy is already very high, then this metric may not give you any new insights as interventions would have very little effect on the actual bottleneck.

3. Concept completeness

In the driving test example, the driving evaluator may wonder whether a CBM underperforms compared to a standard black-box model that does not use concepts. This question can be answered using concept completeness [2], a metric that helps determining if the concepts alone provide sufficient information for the model to solve the task when compared to a black box.

Concept completeness1 is defined as the ratio of the task performance of a CBM to that of a black-box model (θ₍b₎):

In the driving test scenario, if the CBM uses concepts like “traffic light color” and “ambulance presence” and achieves a task accuracy that is close to or greater than a black-box model trained directly on raw input (e.g., image pixels), it implies that the concepts provide the necessary information to complete the task. For instance, if the CBM’s accuracy in predicting whether to stop or cross is 95% and the black-box model’s accuracy is 96%, the high concept completeness score shows that interpretability is achieved with minimal loss in task accuracy.

Limitations: High concept completeness does not guarantee that the concepts themselves are fully interpretable or well-defined; a model might achieve high completeness while relying on concepts that are difficult for humans to understand.

Coding practice: quantitative metrics for concept-based models

In this practice, we will evaluate a CBM’s predictive performance, intervention effectiveness, and concept completeness in a simple traffic light scenario where decisions to cross or stop depend on two concepts: the traffic light being green and the presence of an ambulance. The model predicts these concepts and uses them to make decisions.

⚠️ Warning: This section assumes familiarity with the first chapter on CBMs and PyC (PyC must be installed before running this code). If you’re new to these concepts, it’s recommended to start with this introductory chapter for foundational understanding.

Step #1: Train a CBM

We begin this practice by re-training the Concept Bottleneck Model (CBM) used in the previous coding exercise.

import numpy as np
import torch

from torch_concepts.nn import LinearConceptLayer
from torch_concepts.data import TrafficLights

# Fix seeds first
seed = 42
torch.manual_seed(seed)
np.random.seed(seed)

n_samples = 500

# Loading training dataset
dataset = TrafficLights(
    n_samples=n_samples,
    possible_starting_directions=['west'],
    resize_final_image=0.05,
    selected_concepts=[
        'green light on selected lane',
        'car in intersection',
        'ambulance seen',
        'ambulance approaching perpendicular to selected car',
    ],
    split='train',
)
concept_names, task_names = dataset.concept_names, dataset.task_names
n_concepts = len(concept_names)

# Loading testing dataset
# Generate the test dataset
test_dataset = TrafficLights(
    n_samples=n_samples,
    possible_starting_directions=['west'],
    resize_final_image=0.05,
    selected_concepts=[
        'green light on selected lane',
        'car in intersection',
        'ambulance seen',
        'ambulance approaching perpendicular to selected car',
    ],
    split='test',
)
print(
    f"Training set has {len(dataset)} samples while test set "
    f"has {len(test_dataset)} samples"
)

Output:

Training set has 300 samples while test set has 100 samples

# Define the CBM

latent_dims = 32

# The encoder extracts a low-dimensional representation of the input
encoder = torch.nn.Sequential(
    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(3, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 5x5 max pooling layer
    torch.nn.MaxPool2d((5, 5)),

    # Finally, we flatten and map it to a known latent space size
    torch.nn.Flatten(start_dim=1, end_dim=-1),
    torch.nn.Linear(576, latent_dims),  # 576 comes from the size after flattening
    torch.nn.LeakyReLU(),
)
# The concept scorer predicts concept logits {traffic light color, ambulance presence}
c_layer = LinearConceptLayer(
    in_features=latent_dims,
    out_annotations=concept_names,
)

# The task predictor determines the value of the downstream label {cross}
y_predictor = torch.nn.Sequential(
    torch.nn.Linear(n_concepts, latent_dims),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(latent_dims, 1), # output shape 1 as it is a binary task
)
model = torch.nn.Sequential(encoder, c_layer, y_predictor)

# Print it out
model

Output:

Sequential(
  (0): Sequential(
    (0): Conv2d(3, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
    (1): LeakyReLU(negative_slope=0.01)
    (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
    (3): LeakyReLU(negative_slope=0.01)
    (4): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
    (6): LeakyReLU(negative_slope=0.01)
    (7): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
    (8): LeakyReLU(negative_slope=0.01)
    (9): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (10): MaxPool2d(kernel_size=(5, 5), stride=(5, 5), padding=0, dilation=1, ceil_mode=False)
    (11): Flatten(start_dim=1, end_dim=-1)
    (12): Linear(in_features=576, out_features=32, bias=True)
    (13): LeakyReLU(negative_slope=0.01)
  )
  (1): LinearConceptLayer(
    (transform): Sequential(
      (0): Linear(in_features=32, out_features=4, bias=True)
      (1): Unflatten(dim=-1, unflattened_size=[4])
      (2): Annotate()
    )
  )
  (2): Sequential(
    (0): Linear(in_features=4, out_features=32, bias=True)
    (1): LeakyReLU(negative_slope=0.01)
    (2): Linear(in_features=32, out_features=1, bias=True)
  )
)

We train the model:

from torch.utils.data import DataLoader

n_epochs = 20
concept_loss_weight = 10
lr = 0.01
batch_size = 50

# Define optimizer and loss function
model = torch.nn.Sequential(encoder, c_layer, y_predictor)
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
loss_fn = torch.nn.BCELoss()

# Make a batch dataset loader
dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=True,
    drop_last=False,
    num_workers=2,
)

# Standard PyTorch learning cycle
model.train()
for epoch in range(n_epochs):
    for batch_idx, (x, y, c, _, _) in enumerate(dataloader):
        # Encode input, then predict concept and downstream tasks activations
        emb = encoder(x)
        c_pred = c_layer(emb).sigmoid()
        y_pred = y_predictor(c_pred).sigmoid().view(-1)

        # Double loss on concepts and tasks
        loss = loss_fn(y_pred, y) + concept_loss_weight * loss_fn(c_pred, c)

        # Perform the update
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        task_acc = torch.mean(((y_pred > 0.5) == y).type(torch.float))
        task_acc = task_acc.detach().cpu().numpy()
        if ((epoch + 1) % 5 == 0) and (batch_idx == 0):
            print(
                f"Epoch [{epoch+1}/{n_epochs}], "
                f"Step [{batch_idx+1}/{len(dataloader)}], "
                f"Loss: {loss.item():.4f}, "
                f"Task Accuracy: {task_acc * 100:.2f}%, "
            )

Output:

Epoch [5/20], Step [1/6], Loss: 3.0958, Task Accuracy: 96.00%, 
Epoch [10/20], Step [1/6], Loss: 1.9020, Task Accuracy: 90.00%, 
Epoch [15/20], Step [1/6], Loss: 1.8252, Task Accuracy: 90.00%, 
Epoch [20/20], Step [1/6], Loss: 0.8952, Task Accuracy: 94.00%,

We then evaluate this trained model on the test dataset. For this, we first load the test set into memory (important note: we would only recommend this for very small datasets like the toy example here. With large dataset, you should use batched data loaders instead):

# Load the train set to memory
x_train = []
c_train = []
y_train = []
for (x, y, c, _, _) in dataset:
    x_train.append(x.unsqueeze(0))
    y_train.append(y.unsqueeze(0))
    c_train.append(c.unsqueeze(0))
x_train = torch.concat(x_train, dim=0)
y_train = torch.concat(y_train, dim=0)
c_train = torch.concat(c_train, dim=0)

# Load the test set to memory
x_test = []
c_test = []
y_test = []
for (x, y, c, _, _) in test_dataset:
    x_test.append(x.unsqueeze(0))
    y_test.append(y.unsqueeze(0))
    c_test.append(c.unsqueeze(0))
x_test = torch.concat(x_test, dim=0)
y_test = torch.concat(y_test, dim=0)
c_test = torch.concat(c_test, dim=0)

And we can now evaluate the test concept and label predictions:

model.eval()
c_pred = c_layer(encoder(x_test)).sigmoid()
y_pred = y_predictor(c_pred).sigmoid()
print("Average task prediction:", y_pred.mean(0).detach().cpu().numpy())
print("Average concept prediction:", c_pred.mean(0).detach().cpu().numpy())

Output:

Average task prediction: [0.46639043]
Average concept prediction: [0.74174714 0.5710389  0.32018384 0.2333349 ]

Step #2: Compute task and concept performance

In our traffic light dataset, both concepts (“traffic light color” and “ambulance presence”) and the output (“cross”) are binary variables. This allows us to evaluate task and concept performance using the Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores, a standard metric for binary classification problems.

from sklearn.metrics import roc_auc_score

concept_performance = roc_auc_score(c_test, c_pred.detach())
task_performance = roc_auc_score(y_test, y_pred.detach())

print(f'Task performance: {task_performance*100:.2f}%')
print(f'Concept performance: {concept_performance*100:.2f}%')

Output:

Task performance: 91.62%
Concept performance: 94.39%

After a few epochs, the CBM is already able to accurately predict both concepts and tasks.

Step #3: Compute intervention effectiveness

In our traffic light scenario, we can evaluate how intervening by correcting mispredicted concepts — such as “traffic light color” or “ambulance presence” — improves the model’s decision on whether to “cross.” By replacing predicted concept values with ground truth values and recalculating model performance, we measure how each intervention impacts downstream task accuracy. In PyC, this requires specifying a list of concept index groups to intervene on.

from torch_concepts.metrics import intervention_score

intervention_groups = [[], [0], [1], [0, 1]]

# Evaluate intervention effectiveness of each concept group individually
intervention_scores = intervention_score(
    y_predictor,
    c_pred,
    c_test,
    y_test,
    intervention_groups,
    auc=False,
)
print(f'Individual intervention scores: {intervention_scores}')

# Evaluate the global intervention effectiveness as the AUC
intervention_auc = intervention_score(
    y_predictor,
    c_pred,
    c_test,
    y_test,
    intervention_groups,
)
print(f'Intervention AUC: {intervention_auc:.4f}')

Output:

Individual intervention scores: [0.9161793372319689, 0.9207277452891488, 0.9285250162443145, 0.9285250162443145]
Intervention AUC: 0.9235

The high intervention AUC indicates that concept interventions effectively improve the model’s task performance. Individual intervention scores show that correcting the “traffic light color” concept provides the most significant improvement in downstream task performance.

Step #4: Compute concept completeness

To compute concept completeness, we need a black-box baseline model that uses both raw features and concept labels, as this matches the information provided to the CBM. The following code implements a simple black-box model with a similar parameter count to the CBM for a fair comparison.

# Make maps containing the values of each concept
c_train_maps = c_train.unsqueeze(-1).unsqueeze(-1)
c_train_maps = c_train_maps.expand(-1, -1, 64, 64)
c_test_maps = c_test.unsqueeze(-1).unsqueeze(-1)
c_test_maps = c_test_maps.expand(-1, -1, 64, 64)

# Put them together with the input features
xc_train = torch.concat((x_train, c_train_maps), dim=1)
xc_test = torch.concat((x_test, c_test_maps), dim=1)

# Defining a balck box baseline
baseline = torch.nn.Sequential(
    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(3 + n_concepts, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 5x5 max pooling layer
    torch.nn.MaxPool2d((5, 5)),

    # Finally, we flatten and map it to a known latent space size
    torch.nn.Flatten(start_dim=1, end_dim=-1),
    torch.nn.Linear(576, 1),
)

# Define optimizer and loss function
optimizer = torch.optim.AdamW(baseline.parameters(), lr=lr)
loss_fn = torch.nn.BCELoss()

# Standard PyTorch learning cycle
baseline.train()
for epoch in range(n_epochs):
   optimizer.zero_grad()

   # Encode input, then predict concept and downstream tasks activations
   y_pred_baseline = baseline(xc_train).sigmoid().view(-1)

   # Double loss on concepts and tasks
   loss = loss_fn(y_pred_baseline, y_train)
   loss.backward()
   optimizer.step()

baseline.eval()
y_pred_baseline = baseline(xc_test).sigmoid()
task_performance_baseline = roc_auc_score(y_test, y_pred_baseline.detach())

We can then compute concept completeness as the ratio between the CBM’s downstream task performance and that of the black-box baseline. This ratio indicates the cost of incorporating a concept layer within the network to improve interpretability compared to an equivalent black-box model.

from torch_concepts.metrics import completeness_score

concept_completeness = completeness_score(y_test, y_pred_baseline, y_pred)

print(f'Task performance: {task_performance*100:.2f}%')
print(f'Task performance baseline: {task_performance_baseline*100:.2f}%')
print(f'Concept completeness: {concept_completeness*100:.2f}%')

Output:

Task performance: 91.62%
Task performance baseline: 96.69%
Concept completeness: 94.76%

The resulting concept completeness score suggests that the CBM reflects a moderate drop in downstream task performance compared to the black-box baseline, in exchange for the increased interpretability.

Take Home Message

In summary, this chapter demonstrated how to quantitatively evaluate CBMs using metrics such as task and concept performance, intervention effectiveness, and concept completeness. These metrics provide a insight into the model’s accuracy and interpretability as follows:

• Task and concept performance: This metric quantifies a CBM’s predictive performance, assessing how accurately the model predicts concepts and downstream tasks.
• Intervention effectiveness: This metric quantifies the impact of concepts on a CBM’s task predictions. Specifically, concept interventions can be used to assess how correcting mispredicted concepts can improve downstream task performance.
• Concept completeness: This metric quantifies the cost in downstream task performance of introducing a concept layer compared to a black-box model.

The next chapter will discuss the accuracy-explainability trade-off, a potential challenge in concept-based interpretability, and strategies to mitigate its impact on CBMs.

References

If you found this chapter useful for your blog post, research article or product, we would be grateful if you would cite it like this:

Barbiero P., Ciravegna G., Debot D., Diligenti M., 
Dominici G., Espinosa Zarlenga M., Giannini F., Marra G. (2024).
Concept-based Interpretable Deep Learning in Python.
https://pyc-team.github.io/pyc-book/intro.html

Or use the following bibtex entry:

@book{pycteam2024concept,
  title      = {Concept-based Interpretable Deep Learning in Python},
  author     = {Pietro Barbiero, Gabriele Ciravegna, David Debot, Michelangelo Diligenti, Gabriele Dominici, Mateo Espinosa Zarlenga, Francesco Giannini, Giuseppe Marra},
  year       = {2024},
  url        = {https://pyc-team.github.io/pyc-book/intro.html}
}

Bibliography

[1] Koh, Pang Wei, et al. “Concept bottleneck models.” International conference on machine learning. PMLR, 2020.

[2] Yeh, Chih-Kuan, et al. “On completeness-aware concept-based explanations in deep neural networks.” Advances in neural information processing systems 33 (2020): 20554–20565.

In the “The tale of the deep learning model that failed my driving exam”, we compared the decisions of an 18-year-old driver and a deep learning (DL) model when facing a tricky intersection where the driver had a green light, but an ambulance unexpectedly crossed.

While both drivers stopped at the intersection, the teenager could explain how she arrived at that conclusion in terms of “CONCEPTS” like the light color and the presence (or absence) of the ambulance:

“I would have crossed if the ambulance wasn’t there, but I know for sure that I would never cross in the presence of an ambulance.” [Example of a human driver explanation]

In contrast, the DL model, despite making the correct decision and having been trained on millions of instances where lights and ambulances were around, lacked a satisfying decision-making process for the driving evaluator, as the model’s explanations were based on raw pixel activations:

“If pixel 2890 had an RGB value of (28, 178, 111), the model would have chosen to cross.” [Example of a DL driver explanation]

This example highlights the need for developing DL models with a transparent decision-making process like the teenager’s, especially in high-stakes fields such as medicine (deciding whether to give a certain treatment), finance (deciding whether to approve a loan), and law (checking whether a hiring system is fair).

Blueprint for concept-based deep learning models

⚠️ Warning: The following paragraphs assume a basic understanding of probability.

From a technical standpoint, the problem can be described as follows: we aim to model a relationship between a set of random variables 𝑋 ∈ 𝒳 representing low-level perceptive features (such as pixels) and a set of random variables corresponding to decisions 𝑌 ∈ 𝒴 (whether to cross or stop).

The DL model in the tale modeled the relationship as p(𝑌 = ‘cross’ ∣ 𝑋 = ‘image’s pixels’), directly mapping raw image data to a decision.

The teenager, instead, modeled the same problem using intermediate higher-level variables — referred to as “concepts”, like the light color and the ambulance — leading to a more human-interpretable decision-making process. Based on such concepts, her reasoning could be expressed as p(𝑌 = “cross” ∣ 𝐶₁ = “light color”, 𝐶₂ = “ambulance”), where the concepts 𝐶 ∈ 𝒞 provided insight into how she made a particular decision.

To emulate the teenager’s approach, we re-write the probability distribution p(Y ∣ X) as if it were obtained by marginalizing the joint distribution p(Y, C ∣ X) over the concepts C. Now, assuming that Y is conditionally independent of X given C, we can obtain the following factorization:

In this formulation, the model observes the input features X (such as pixels from an image) and maps them to a set of interpretable, high-level variables C, known as “concepts” [1,2,3]. These concepts are analogous to the reasoning elements identified by the teenager — such as the traffic light color or the presence of an ambulance. The second part of the model then uses these concepts to determine the decision Y (whether to cross or stop). This class of models is known as a Concept Bottleneck Model (CBM) [4]. CBMs parametrize the conditional distributions with a pair of neural networks:

• A “concept encoder” 𝑔(⋅) that takes as an input a sample 𝐱 ∈ 𝒳, and predicts a set of k concepts 𝐜 ∈ 𝒞. This network parameterizes the concept distribution.
• A “task predictor” 𝑓(⋅) that takes as an input a concept vector 𝐜 ∈ 𝒞 and predicts an output label 𝓎 ∈ 𝒴. This network parameterizes the output distribution.

Given a concept-based dataset of i.i.d. triples (input, concepts, task) 𝐷 = {(𝐱, 𝐜, 𝓎)}, the CBM’s parameters (θ₍g₎ and θ₍f₎) are usually optimized via gradient descent by maximizing the empirical log-likelihood of the training data:

In most instances (but not all), we are trying to model a scenario where we are given 𝑘 different concept annotations during training and the output task can be one of 𝐿 different labels (e.g., multi-task classification). In that case, it is common to model the data distribution by interpreting the output of 𝑔 and 𝑓 as probabilities over their output domains. That is

1. The concept encoder 𝑔(𝑥) = ĉ ∈ [0, 1]^𝑘 outputs a 𝑘-dimensional vector where ĉᵢ = p(𝑐ᵢ = 1 ∣ 𝑋 = 𝑥), and
2. The label predictor 𝑓(𝑐) = ŷ ∈ [0, 1]^𝐿 outputs an 𝐿-dimensional vector where 𝐿 is the number of output labels, ∑ᵢ ŷᵢ = 1 (e.g., a softmax activation is used), and ŷᵢ = p(𝑌 = i ∣ 𝐶 = 𝑐).

In this instance, the likelihood above can be expressed as:

Where ℒ₍CE₎ and ℒ₍BCE₎ stand for the standard cross entropy and binary cross entropy losses (notice the arg max turned into an arg min as we are working with negative log-likelihoods when using the cross entropy losses).

In CBMs there are three different ways to optimize this objective function:

1. Independent training optimizes f and g independently: f is trained using ground truth concepts c as input. At test time, f takes g(x) as input.
2. Sequential training first optimizes g independently. Once trained, the parameters of the concept encoder g are frozen, and f is trained using g(x) as input.
3. Joint training optimizes f and g at the same time and f takes as input g(x).

The following coding practice introduces you to implementing CBMs and shows how to query CBMs to understand the model’s decision-making process.

Coding practice

In this practice, we implement a Concept Bottleneck Model (CBM) for a simple traffic light scenario where decisions to cross or stop depend on two concepts: the traffic light being green and the presence of an ambulance. The model predicts these concepts and uses them to make decisions.

⚠️ Warning: This section assumes a basic understanding of programming machine learning scripts using deep learning frameworks, specifically PyTorch. If you’re new to PyTorch, consider starting with these introductory tutorials to get up to speed.

Step #1: Install PyC

First, we install the necessary Python packages required to implement CBMs. This includes our library pytorch-concepts which runs on top of standard deep learning libraries (PyTorch and torch_geometric).

pip install torch
pip install -i https://test.pypi.org/simple/ --upgrade pytorch-concepts

Step #2: Load traffic scenario

Next, we load the dataset for our traffic light scenario. This toy synthetic dataset, programmatically generated in the call below, comprises a collection of images of a single intersection under different conditions. Each image is an image of an intersection where the “target” or “selected” car (the car we need to make the decision for) is approaching the intersection from the left (moving in a west-to-east direction). The task is to decide whether this car should “stop” (label is 0) or “continue” (task label is 1).

To help with learning concept-based models, this dataset’s images are annotated with the following concept labels:

• Green Light on Selected Lane: Indicates whether the traffic light on the lane of the selected car is green or not.
• Car in Intersection: Specifies whether another car is visible in the intersection.
• Ambulance Seen: Specifies whether an ambulance is visible in the scene.
• Ambulance Approaching Perpendicular to Selected Car: Specifies whether an ambulance is approaching in a north-to-south or south-to-north direction (i.e., it is moving perpendicularly with respect to the target car).

# Making seeds fixed for determinism

import numpy as np
import torch

seed = 42
print(f"Setting random seeds to {seed}")
torch.manual_seed(seed)
np.random.seed(seed)

from torch_concepts.data import TrafficLights

# In practice, we need datasets that are significantly larger, but we will
# not use those for this example as we want to focus on the main ideas rather
# than on performane
n_samples = 500

# Loading dataset
dataset = TrafficLights(
    n_samples=n_samples,
    possible_starting_directions=['west'],
    resize_final_image=0.05,
    selected_concepts=[
        'green light on selected lane',
        'car in intersection',
        'ambulance seen',
        'ambulance approaching perpendicular to selected car',
    ],
    split='train',
)
concept_names, task_names = dataset.concept_names, dataset.task_names
n_concepts = len(concept_names)

# Example of scenario
print(f"Input shape: {dataset.input_shape}")
print(f"Using {n_concepts} concepts with semantics: {concept_names}")
print(f"Task has {len(task_names)} downstream tasks: {task_names}")

Output:

Input shape: (3, 64, 64)
Using 4 concepts with semantics: ['green light on selected lane', 'car in intersection', 'ambulance seen', 'ambulance approaching perpendicular to selected car']
Task has 1 downstream tasks: ['continue']

We can see some of examples of this dataset to get a sense as to how they look:

import matplotlib.pyplot as plt
import random

# Select a random sample of indices for the grid
grid_size = (3, 3)
indices = random.sample(range(len(dataset)), grid_size[0] * grid_size[1])
sampled_images = [dataset[i][0] for i in indices]
sampled_labels = [dataset[i][1] for i in indices]
label_names = ['stop', 'continue']

# Plot images in a grid
fig, axes = plt.subplots(grid_size[0], grid_size[1], figsize=(10, 10))
axes = axes.flatten()  # Flatten to easily iterate over axes if needed

for i, ax in enumerate(axes):
    ax.imshow(
        sampled_images[i].detach().cpu().numpy().transpose(1, 2, 0),
        cmap=None,
    )
    ax.set_title(
        f"Label: {label_names[int(sampled_labels[i])]}",
        fontsize=10,
    )
    ax.axis('off')  # Hide axes

plt.tight_layout()
plt.show()

Step #3: Define the CBM

We are now ready to construct our first CBM. The model comprises three main components: encoder, concept predictor, and task predictor. These components are combined sequentially to form the complete CBM architecture.

Encoder: The encoder takes as input low-level features 𝑋 and produces a lower-dimensional representation. The specific architecture of the encoder depends on the input data type; it could be a convolutional neural network, a recurrent neural network, a transformer, or another suitable model.

Given that we will be working with images, we will use a very simple Convolutional Neural Network (CNN):

import torch

latent_dims = 32

# The encoder extracts a low-dimensional representation of the input
encoder = torch.nn.Sequential(
    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(3, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 3x3 convolution with 4 output channels
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),

    # A 3x3 convolution with 4 output channels with a batch norm
    torch.nn.Conv2d(4, 4, (3, 3), padding='same'),
    torch.nn.LeakyReLU(),
    torch.nn.BatchNorm2d(4),

    # A 5x5 max pooling layer
    torch.nn.MaxPool2d((5, 5)),

    # Finally, we flatten and map it to a known latent space size
    torch.nn.Flatten(start_dim=1, end_dim=-1),
    torch.nn.Linear(576, latent_dims),  # 576 comes from the size after flattening
    torch.nn.LeakyReLU(),
)
encoder

Output:

Sequential(
  (0): Conv2d(3, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (1): LeakyReLU(negative_slope=0.01)
  (2): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (3): LeakyReLU(negative_slope=0.01)
  (4): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (5): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (6): LeakyReLU(negative_slope=0.01)
  (7): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (8): LeakyReLU(negative_slope=0.01)
  (9): BatchNorm2d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (10): MaxPool2d(kernel_size=(5, 5), stride=(5, 5), padding=0, dilation=1, ceil_mode=False)
  (11): Flatten(start_dim=1, end_dim=-1)
  (12): Linear(in_features=576, out_features=32, bias=True)
  (13): LeakyReLU(negative_slope=0.01)
)

Linear Concept Layer: A linear concept layer takes an embedding as input and generates a set of concept representations, which in this case correspond to concept logits. To implement concept layers, we use the PyC library.

To define this layer, the following inputs are required:

• in_features: The size of the embedding generated by the encoder.
• annotations: A list of integers or a list where each entry represents an axis of the output concept tensor, and, if it is a list, each element in that list represents the semantics assigned to the neuron at that corresponding index for that dimension.

For example, we can make a LinearConceptLayer that takes as an input a latent representation of size 8 and outputs two neurons aligned to concepts ["ambulance", "green light"] through the following calls:

c_layer_annotated = LinearConceptLayer(
    in_features=2,
    out_annotations=["ambulance", "green light"],  # If we wish to annotate the output neurons with known concepts
)

or

c_layer_unannotated = LinearConceptLayer(
    in_features=2,
    out_annotations=2,  # If we do not wish to annotate the output neurons
)

In our example, this layer produces two concept logits corresponding to “traffic light green” and “ambulance crossing”.

from torch_concepts.nn import LinearConceptLayer

# The concept scorer predicts concept logits {traffic light color, ambulance presence}
c_layer = LinearConceptLayer(
    in_features=latent_dims,
    out_annotations=concept_names,
)
print('Architecture:\n', c_layer)
print('Expected concepts in output tensor:\n', c_layer.annotations)

Output:

Architecture:
 LinearConceptLayer(
  (transform): Sequential(
    (0): Linear(in_features=32, out_features=4, bias=True)
    (1): Unflatten(dim=-1, unflattened_size=[4])
    (2): Annotate()
  )
)
Expected concepts in output tensor:
 [['green light on selected lane', 'car in intersection', 'ambulance seen', 'ambulance approaching perpendicular to selected car']]

In PyC concept layers produce as output an object of type AnnotatedTensor whose dimensions are labeled with the provided concept names:

concept_tensor = c_layer(torch.randn(1, latent_dims))
print(concept_tensor)

Output:

AnnotatedTensor of shape torch.Size([1, 4]), dtype torch.float32, and annotations [None, ['green light on selected lane', 'car in intersection', 'ambulance seen', 'ambulance approaching perpendicular to selected car']] for each dimension.

The object AnnotatedTensor has built-in methods to extract concept representations from specific dimensions which might be practical for in-depth analysis:

concept_traffic_light = concept_tensor.extract_by_annotations(['green light on selected lane'])
print(concept_traffic_light)

Output:

AnnotatedTensor of shape torch.Size([1, 1]), dtype torch.float32, and annotations [None, ['green light on selected lane']] for each dimension.

Task Predictor: Takes as input the predicted concept representations to determine the final decision on whether to cross or not. In this case, we implement the task predictor with a multi-layer perceptron, but we will discuss more advanced (and interpretable) task predictors later on in this series.

# The task predictor determines the value of the downstream label {cross}
y_predictor = torch.nn.Sequential(
    torch.nn.Linear(n_concepts, latent_dims),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(latent_dims, 1), # output shape 1 as it is a binary task
)
y_predictor

Output:

Sequential(
  (0): Linear(in_features=4, out_features=32, bias=True)
  (1): LeakyReLU(negative_slope=0.01)
  (2): Linear(in_features=32, out_features=1, bias=True)
)

Step #4: Train the CBM

We set the training parameters, including the number of epochs (n_epochs), the batch size (batch_size) and learning rate (lr). Using the AdamW optimizer and Binary Cross-Entropy Loss, we train the CBM through a standard PyTorch loop. Here we will train the model using its joint training, where the concept encoder and the label predictor are trained in an end-to-end manner.

Specifically, in each epoch the model performs a forward pass to predict concepts and the final decision, computes the combined loss, performs backpropagation, and updates the model parameters. The concept regularization weight (concept_loss_weight) controls the weight of the concept loss w.r.t. the downstream task loss:

from torch.utils.data import DataLoader

n_epochs = 20
concept_loss_weight = 10
lr = 0.01
batch_size = 50

# Define optimizer and loss function
model = torch.nn.Sequential(encoder, c_layer, y_predictor)
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
loss_fn = torch.nn.BCELoss()

# Make a batch dataset loader
dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=True,
    drop_last=False,
    num_workers=2,
)

# Standard PyTorch learning cycle
model.train()
for epoch in range(n_epochs):
    for batch_idx, (x, y, c, _, _) in enumerate(dataloader):
        # Encode input, then predict concept and downstream tasks activations
        emb = encoder(x)
        c_pred = c_layer(emb).sigmoid()
        y_pred = y_predictor(c_pred).sigmoid().view(-1)

        # Double loss on concepts and tasks
        loss = loss_fn(y_pred, y) + concept_loss_weight * loss_fn(c_pred, c)

        # Perform the update
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        task_acc = torch.mean(((y_pred > 0.5) == y).type(torch.float))
        task_acc = task_acc.detach().cpu().numpy()
        if ((epoch + 1) % 5 == 0) and (batch_idx == 0):
            print(
                f"Epoch [{epoch+1}/{n_epochs}], "
                f"Step [{batch_idx+1}/{len(dataloader)}], "
                f"Loss: {loss.item():.4f}, "
                f"Task Accuracy: {task_acc * 100:.2f}%, "
            )

Output:

Epoch [5/20], Step [1/6], Loss: 3.3699, Task Accuracy: 82.00%, 
Epoch [10/20], Step [1/6], Loss: 1.8929, Task Accuracy: 96.00%, 
Epoch [15/20], Step [1/6], Loss: 1.2713, Task Accuracy: 94.00%, 
Epoch [20/20], Step [1/6], Loss: 0.6090, Task Accuracy: 100.00%,

Step #5: Trace task prediction back to concept activations

We can examine how the concept activations (green light and ambulance presence) influence the downstream task prediction.

For example, let’s look at a sample where there is a green light for the car, there are no ambulances, and no cars in the intersection. For this, we will first find an example in the training set that has all these properties:

# Load the data to memory for easy use (we would not recommend doing this for
# large datasets!):
x_samples = []
c_samples = []
y_samples = []
for (x, y, c, _, _) in dataset:
    x_samples.append(x.unsqueeze(0))
    y_samples.append(y.unsqueeze(0))
    c_samples.append(c.unsqueeze(0))
x_samples = torch.concat(x_samples, dim=0)
y_samples = torch.concat(y_samples, dim=0)
c_samples = torch.concat(c_samples, dim=0)

import numpy as np

# Find all training examples where the light is green, there is no ambulance,
# and there are no cars in the intersection:
samples_with_green_light = (c_samples[:, 0] == 1).detach().cpu().numpy()
samples_with_int_cars = (c_samples[:, 1] == 1).detach().cpu().numpy()
samples_with_ambulance = (c_samples[:, 2] == 1).detach().cpu().numpy()
samples_with_perp_ambulance = (c_samples[:, 3] == 1).detach().cpu().numpy()
selected_samples = (
    samples_with_green_light &
    np.logical_not(samples_with_int_cars) &
    np.logical_not(samples_with_ambulance) &
    np.logical_not(samples_with_perp_ambulance)
)
print(
    "Found",
    np.sum(selected_samples),
    "samples with green light, no ambulance, and no cars on the intersection"
)

Output:

Found 82 samples with green light, no ambulance, and no cars on the intersection

import matplotlib.pyplot as plt

# Select one of those examples for visualization:
selected_sample = x_samples[selected_samples, :, :, :][0]
selected_label = y_samples[selected_samples][0]
label_names = ['stop', 'continue']

# And visualize it
plt.imshow(
    selected_sample.detach().cpu().numpy().transpose(1, 2, 0),
    cmap=None,
)
plt.title(
    f"Label: {label_names[int(selected_label)]}",
    fontsize=10,
)
plt.axis('off')

plt.tight_layout()
plt.show()

Now, let’s look at our model’s predictions for this sample:

model.eval()
emb = encoder(selected_sample.unsqueeze(0))
c_pred = c_layer(emb).sigmoid()
y_pred = y_predictor(c_pred).sigmoid().view(-1)
print(f"Task Prediction: {label_names[(y_pred[0]>0.5)]}")
for idx, concept_name in enumerate(concept_names):
    pred_str = "ON" if c_pred[0, idx] > 0.5 else "OFF"
    print(
        f"\tOur model predicted concept '{concept_name}' to be {pred_str} "
        f"(with prob {c_pred[0, idx]})"
    )

Output:

Task Prediction: continue
 Our model predicted concept 'green light on selected lane' to be ON (with prob 0.9999604225158691)
 Our model predicted concept 'car in intersection' to be OFF (with prob 0.01809137873351574)
 Our model predicted concept 'ambulance seen' to be OFF (with prob 3.915355136996368e-06)
 Our model predicted concept 'ambulance approaching perpendicular to selected car' to be OFF (with prob 1.5054481082188431e-05)

The model correctly identifies that the traffic light is green and there is no ambulance or cars in the intersection! This can be used as an explanation for why the model decided the car, coming from the left, to cross (task prediction is continue).

Step #6: Change concept activations to affect task predictions

Finally, we alter the concept activations to different values and observe how these changes influence the downstream task prediction. This demonstrates how the model’s decisions are directly tied to specific concept activations.

# Intervene changing the value of the concepts "ambulance" and
# "ambulance approaching perpendicular to selected car" to True
c_intervened = c_pred[0].clone().unsqueeze(0)
c_intervened[0, 2] = 1 # Ambulance set to True
c_intervened[0, 3] = 1 # Ambulance in perpendicular direction set to True

# Compute new task prediction
y_intervened = y_predictor(c_intervened).sigmoid()

print(f"Task Prediction: {label_names[(y_intervened[0]>0.5)]}")
for idx, concept_name in enumerate(concept_names):
    pred_str = "ON" if c_intervened[0, idx] > 0.5 else "OFF"
    print(
        f"\tOur model predicted concept '{concept_name}' to be {pred_str} "
        f"(with prob {c_intervened[0, idx]})"
    )

Output:

Task Prediction: stop
 Our model predicted concept 'green light on selected lane' to be ON (with prob 0.9999604225158691)
 Our model predicted concept 'car in intersection' to be OFF (with prob 0.01809137873351574)
 Our model predicted concept 'ambulance seen' to be ON (with prob 1.0)
 Our model predicted concept 'ambulance approaching perpendicular to selected car' to be ON (with prob 1.0)

The model effectively reacts to the concept intervention1 predicting that, in the presence of a greenlight and of an ambulance in the direction perpendicular to it, the car should stop (task prediction is 0).

More interestingly, pehaps, we can look at what happens when there is an ambulance, but it is moving on the same axis as the target car (and therefore we wouldn’t expect it to have to stop at a green light):

# Intervene changing the value of the concepts "ambulance" and
# "ambulance approaching perpendicular to selected car" to False
c_intervened = c_pred[0].clone().unsqueeze(0)
c_intervened[0, 2] = 1 # Ambulance set to True
c_intervened[0, 3] = 0 # Ambulance in perpendicular direction set to False

# Compute new task prediction
y_intervened = y_predictor(c_intervened).sigmoid()

print(f"Task Prediction: {label_names[(y_intervened[0]>0.5)]}")
for idx, concept_name in enumerate(concept_names):
    pred_str = "ON" if c_intervened[0, idx] > 0.5 else "OFF"
    print(
        f"\tOur model predicted concept '{concept_name}' to be {pred_str} "
        f"(with prob {c_intervened[0, idx]})"
    )

Output:

Task Prediction: continue
 Our model predicted concept 'green light on selected lane' to be ON (with prob 0.9999604225158691)
 Our model predicted concept 'car in intersection' to be OFF (with prob 0.01809137873351574)
 Our model predicted concept 'ambulance seen' to be ON (with prob 1.0)
 Our model predicted concept 'ambulance approaching perpendicular to selected car' to be OFF (with prob 0.0)

Here we see that the model decides to continue, providing further evidence that the model is indeed properly capturing the expected reasoning for this setup!

Take home message

In summary, in this chapter we demonstrated how to implement Concept Bottleneck Models (CBMs). These models are inherently explainable as:

• Task predictions can be traced back to the activation of human-interpretable concepts. This enables the model to answer the driving evaluator’s question by saying, “I decided to cross as I saw a green light and there was no ambulance”.
• Altering concept values changes the model’s decisions. This allows the model to respond to the evaluator’s question with, “In the same scenario, if there is an ambulance, I would cross”.

The next chapter will discuss key metrics used in concept-based interpretability to quantitatively evaluate CBM performance.

References

If you found this chapter useful for your blog post, research article or product, we would be grateful if you would cite it like this:

Barbiero P., Ciravegna G., Debot D., Diligenti M., 
Dominici G., Espinosa Zarlenga M., Giannini F., Marra G. (2024). 
Concept-based Interpretable Deep Learning in Python. 
https://pyc-team.github.io/pyc-book/intro.html

Or use the following bibtex entry:

@book{pycteam2024concept,
  title      = {Concept-based Interpretable Deep Learning in Python},
  author     = {Pietro Barbiero, Gabriele Ciravegna, David Debot, Michelangelo Diligenti, Gabriele Dominici, Mateo Espinosa Zarlenga, Francesco Giannini, Giuseppe Marra},
  year       = {2024},
  url        = {https://pyc-team.github.io/pyc-book/intro.html}
}

Bibliography

[1] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6541–6549. 2017.

[2] Ruth Fong and Andrea Vedaldi. Net2vec: quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 8730–8738. 2018.

[3] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and others. Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). In International conference on machine learning, 2668–2677. PMLR, 2018.

[4] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, 5338–5348. PMLR, 2020.

It’s exam day, and you are the driving evaluator. Two very different “drivers” are up for their final test, and it’s your job to determine if they’re ready for a driving license. The first candidate is a teenager, Sarah — she’s about 18, a little nervous but focused. The second “driver” is not a person at all; it’s a deep learning model based on a massive Transformer architecture, trained to predict driving actions like turning, braking, and accelerating based on live HD camera input.

You take both out on a pre-defined driving circuit, where they’ll need to handle basic maneuvers. But there’s one test you’re particularly interested in — a tricky intersection with a catch: as they approach, the traffic light is green, but just as they get closer, an ambulance speeds through and crosses the intersection.

Driving test scenario: the traffic light is green, but just as the driver gets closer, an ambulance speeds through and crosses the intersection.

You ride with Sarah first. As you approach the intersection, the ambulance cuts across. Sarah reacts quickly, easing the car to a stop. You’re pleased — she’s cautious, and made the right call.

Next, you get into the vehicle controlled by the deep learning model. The system, connected to the car’s camera, processes the scene in real time. It, too, stops smoothly as the ambulance passes, seemingly making the correct decision. You feel a sense of relief — both drivers handled the situation perfectly.

However, you are a stickler for details. Stopping was the right move, but did they both do it for the right reasons? Was it understanding or just luck? To find out, you turn to your post-drive evaluation questions, aimed at revealing the drivers’ thought processes.

You start with Sarah. “Can you explain to me why you decided to stop?” you ask.

“I saw the green light, and I was ready to cross,” she begins. “But then the ambulance appeared. I would have crossed if the ambulance wasn’t there, but I know for sure that I would never cross in the presence of an ambulance.” Her response is clear and confident.

“I would have crossed if the ambulance wasn’t there, but I know for sure that I would never cross in the presence of an ambulance” [Example of human explanation]

Satisfied with Sarah’s reasoning, you grant her the driving license. She clearly understood the situation and could articulate her decision-making process. She’ll be a cautious and responsible driver.

Now it’s the deep learning model’s turn. You turn to your AI expert friend for help in interpreting the model’s behavior. “Can you explain to me how the model decided to stop?” you ask.

Your friend accesses the system and provides an explanation: “When the mean activation of input embedding block 42 exceeds 173, the model usually chooses to stop. However, in this case, if pixel 2890 had an RGB value of (28, 178, 111), the model would have chosen to cross instead”.

“If pixel 2890 had an RGB value of (28, 178, 111), the model would have chosen to cross” [Example of deep learning explanation]

You frown, confused. What does that have to do with stopping for an ambulance? The explanation feels disconnected from the real-world reasoning you’re used to hearing. The model’s response is technical, tied to pixel values and activations, rather than an understanding of traffic rules or emergency vehicles. It stopped, yes, but did it truly understand why?

As you sit there, contemplating, the question lingers in your mind: would you give the deep learning model a driving license?

Series overview

This series will cover the following chapters:

• Concept Bottleneck Models: an introduction to concept-based interpretable architectures.
• Quantitative metrics in concept interpretability: analysis of the most commonly used metrics in concept based interpretability.
• … and more coming soon!

Conference highlights

The PyC team will also cover highlights from top-tier AI conferences on concept-based interpretability:

NeurIPS’24

• Interpretable Concept-based Memory Reasoning: accurate, interpretable, verifiable deep learning models
• Relational Concept Bottleneck Models: Graph Neural Networks expressivity + concept-based interpretability

If you would like to include your poster in this track, please contact us.

References

If you found this series useful for your blog post, research article or product, we would be grateful if you would cite it like this:

Barbiero P., Ciravegna G., Debot D., Diligenti M., 
Dominici G., Espinosa Zarlenga M., Giannini F., Marra G. (2024). 
Concept-based Interpretable Deep Learning in Python. 
https://pyc-team.github.io/pyc-book/intro.html

Or use the following bibtex entry:

@book{pycteam2024concept,
  title      = {Concept-based Interpretable Deep Learning in Python},
  author     = {Pietro Barbiero, Gabriele Ciravegna, David Debot, Michelangelo Diligenti, Gabriele Dominici, Mateo Espinosa Zarlenga, Francesco Giannini, Giuseppe Marra},
  year       = {2024},
  url        = {https://pyc-team.github.io/pyc-book/intro.html}
}

Problem statement

Understanding and taking decisions about complex scenes are very challenging tasks, for both humans and AI. For instance, imagine you are the evaluator at the driving theory exam (see also the post on practice exam!), and you are comparing the answers of a human (Bob) and two AI agents (lets call them Ag1 and Ag2). By checking their answers, you stumbled upon two apparently similar scenarios, requiring to predict the expected behavior of drivers at a crossing. Your curiosity is triggered when you notice that one AI agent provides different responses for the two cases.

First question: “Is it true that vehicle H should be the first to pass in Figure 1?”

Looking at the responses, you notice that all (humans and AI) candidates correctly answered. However, as you are a meticulous evaluator, you ask them to motivate their answers. Bob correctly explains that the rules of the road state that vehicles should give priority to the right, and for this reason, H should pass first, as it is the only vehicle with no other vehicles on its right. Ag1, whose decision mechanism is based on a Concept Bottleneck Model (CBM) [1], similarly explains that H should pass as it has no other vehicles on its right. Finally Ag2, whose decision mechanism is based on a Graph Neural Network (GNN) [2], has also answered corrected, but it explains the reason in this way “The activation of input embedding block 3 exceeds 100, thus the prediction is that H should pass first”.

You’re quite convinced about the rationale behind the answers of Bob and Ag1, however you become very skeptical about accepting the correctness of Ag2’s response. You could have stopped here the inquiry, however as you are actually very meticulous, you decide also to ask about the answers to the other question, the one referring to the scenario in Figure 2.

Second question: “Is it true that vehicle B should go before vehicle O , but after that vehicles L and H pass together in Figure 2?”

Even if similar, the two pictures differentiate for a crucial detail, the presence of yields’ signs. Looking at the responses, you notice that Bob and Ag2 still correctly answered, whereas this time Ag1 is wrong. You already know that Ag2 explanations are almost unreadable, thus you ask just Bob and Ag1 to motivate their answers. Bob correctly explains that a yield sign is more important than naively giving priority to right for vehicles, and for this reason, the correct passing order in Figure 2 is: L and H first, which can go together as they are not affecting the priority of each other, then B and finally O , as B is at the right of O. When you ask to Ag1 why it failed to provide this correct answer, it replicates that it knows for sure that B should go before vehicle O , but it has no idea on how to decide if L and H can go together, thus it just flipped a coin, and answered NO. The reason why Ag1 was not able to answer with precision is that it can only precisely answer about questions concerning a single object, and it has been not designed to decide if two different cars can go together.

Relational Concept Bottleneck Models

⚠️ Warning: This section is easier to follow with a basic understanding of the First-Order Logic (FOL) mathematical notation.

As we saw from the examples above, while GNNs (like Ag2) are able to correctly take decisions in relational scenarios, they act as black-box and are not capable of explaining their decisions. On the other hand, standard CBMs (like Ag1) are capable of explaining their predictions, but only on decisions involving a single object (i.e. a car) per time.

Goal: “Design DL models capable of relational reasoning (akin to a GNN), while also being interpretable (akin to a CBM)”.

First-Order Logic Representation

A classical CBM would for example say that if a car named car1 has neither another car at its right nor a yield sign, then it can go. Using FOL to express this knowledge, it would be:

However, it would be problematic to describe the collective behavior at the crossing without expressing relational concepts involving multiple cars. Relational Concept Bottleneck Models (R-CBM) merge CBMs and GNNs into an interpretable relational setting with the ability to model the relations among groups of entities.

For example, an R-CBM can express the behavior at the intersection using pairwise priorities rules such as:

• a yield sign for a car results for a higher priority for another incoming car:

• similarly, a red light for a car, results for a higher priority for another incoming car:

• if no red light or yield is present the car at the right has priority:

• if a car has not to give priority to any other car, it has right to go:

Architecture

The overall structure of an R-CBM is very similar to the one of a CBM [1] (consisting in the composition of a concept encoder and a task predictor), however R-CBMs do not make a drastic distinction between concepts and tasks, and predicates can be more than unary, thus potentially ground on tuples of single entities. More specifically, given a relational language with entity and predicate symbols, an R-CBM first processes the atoms like RightOf(car1,car2), Yield(car2) by an encoder, and then map them by a task predictor (like a CBM). The final prediction like Go(car1) is computed by aggregating all the ones from separate groups of neighbour atoms, according to a given dependency graph (like a GNN). While the same task predictors of a CBM can be used for an R-CBM, the atom encoder in R-CBMs can be a bit more complex, as dealing with a relational language. For instance, common choices to embed entity, relations and atoms are given by Knowledge Graph Embeddings [5] or GNNs according to the symbolic or feature-based nature of the input data. We refer to the paper for more technical details.

Relational Deep Concept Reasoner (R-DCR)

In most real scenarios, the rules regulating an environment are hard to codify. This can be due to multiple factors:

• the rules are not known a priori;
• it can be very tedious for a human expert to list all the rules;
• the rules could have a probabilistic nature, and the machine should learn the contexts where a rule applies.

Among R-CBMs, R-DCR is a model allowing to learn the rules determining a decision starting from a weaker input information, in the form of a template determining the factors to consider like RedLight(𝑥),Yield(𝑥),RightOf(𝑥,𝑦). For instance, an R-DCR can be trained by observing the behavior of real drivers at a crossing, e.g. by using data from a camera. An R-DCR’s model processes the atoms in the template using an encoder, decides which subset of atoms are relevant for the final decision and how they should be accounted for. Finally, an R-DCR outputs a logic rule which can be executed to determine the final prediction, like whether which car can go. For example, it could use the template to output the rule:

to express the fact that a car goes if it has no red light and has no other car at its right or the car at its right has a red light. A sketch of the overall computation of an R-DCR for this example is shown in Figure 3.

If interested, please have a look at the full paper or look to our poster at NeurIPS 2024 conference.

Please use the following citation if you want to refer to this work:

@inproceedings{barbierorelational,
  title={Relational Concept Bottleneck Models},
  author={Barbiero, Pietro and Giannini, Francesco and Ciravegna, Gabriele and Diligenti, Michelangelo and Marra, Giuseppe},
  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems}
}

References:

[1] Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020, November). Concept bottleneck models. In International conference on machine learning (pp. 5338–5348). PMLR.
[2] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., & Dahl, G. E. (2020). Message passing neural networks. Machine learning meets quantum physics, 199–214.
[3] Barbiero, P., Giannini, F., Ciravegna, G., Diligenti, M., & Marra, G. Relational Concept Bottleneck Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[4] Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017). Knowledge graph embedding: A survey of approaches and applications. IEEE transactions on knowledge and data engineering, 29(12), 2724–2743.

Problem Statement

Imagine a smart home lighting system that intelligently manages the lights based on sensory data. This system needs to predict when to flick a light switch. The generally desired behavior (Figure 1) can be described using the following two rules that the system must learn:

1. If there is a person and the switch is off, then flick the switch.
2. If there is no person and the switch is on, then flick the switch.

Additionally, the first rule should be ignored in some situations. For instance, during the day, the light should never be flicked on. The model should learn this behavior as well.

To build this system, you could rely on a deep neural network. It would likely perform perfectly, but its decision-making process would be a black box — accurate but not interpretable. Alternatively, you could use a concept-based model (CBM), which explains decisions using human-understandable concepts like “person” and “switch is on.” However, existing CBMs face a trade-off: they are either accurate or fully interpretable, but not both. This is because of three main issues:

1. Limited expressivity. Some approaches are interpretable but achieve this by limiting the model’s expressivity (e.g. Concept Bottleneck Model with a linear layer [1]), which costs accuracy in situations that require non-linear behavior. In our running example, the two rules together encode a relationship that is inherently non-linear. The system must combine the concepts of “person” and “switch is off” for Rule 1, and “no person” and “switch is on” for Rule 2, in a way that resembles an XOR operation: the light should only be flicked if exactly one of these conditions is true (“person” XOR “switch is on”). A linear model cannot capture this kind of non-linear dependency, leading to errors in decision-making.
2. Information bottleneck. Many approaches rely only on the concepts for decision-making (e.g. all Concept Bottleneck Models [1]). This limitation reduces accuracy in scenarios where the concepts alone are insufficient to perfectly predict the task. In our running example, such models lack the possibility to incorporate additional context, like whether it is day or night, because the decision-making is constrained to the predefined concepts. As a result, a model that learned Rule 1 might incorrectly use it during the day to turn the light on, even though it should remain off.
3. Local interpretability. For some approaches, only decisions on individual instances can be explained (i.e. local), but the model’s overall reasoning process remains opaque (e.g. Deep Concept Reasoner [2]). For instance, Deep Concept Reasoner predicts a decision rule over the concepts, which is then evaluated using the concepts to make a decision for the target task. In the context of our running example, this model might predict Rule 1 for a specific input. This provides interpretability for that single decision. However, because the model directly predicts the decision rule using a neural network, all possible rules are encoded implicitly in the model’s weights. This means you cannot explicitly know or verify the full set of decision rules the model might follow, making it impossible to interpret its reasoning globally across all scenarios.

Our work addresses this gap, introducing Concept-Based Memory Reasoner [3], a neurosymbolic concept-based model that is both accurate and globally interpretable. It allows you to understand the system’s reasoning across all scenarios, even enabling formal verification of its rules. This means you can not only trust the system to make the right decisions but also verify and explain why those decisions are made. Figure 2 gives a comparison with the aforementioned models.

Concept-Based Memory Reasoner

Our proposed model, CMR, consists of the following key components:

1. Concept Predictor: Like other concept-based models, CMR uses a neural network to predict human-understandable concepts from the input.
2. Memory of Rules: During training, CMR learns a set of rules that map concepts (like “person” and “switch is on”) to the target task (like “flick the switch”). These rules are stored in a memory, representing all possible reasoning paths the model might use.
3. Rule Selector: Instead of applying all learned rules on the predicted concepts, CMR uses a neural network that acts as a selector. For any given input, this selector chooses a single rule from the memory to make the prediction. This ensures that only the most relevant rule is applied, providing explanations that are concise and task-specific. As the selector takes the image as input (and not just the concepts), it can exploit contextual information that is missing in the concepts (e.g. whether it is day).
4. Symbolic Evaluation: CMR predicts the target task by evaluating the Rule Selector’s chosen rule symbolically using the predicted concepts.

The result is a model with accuracy comparable to black-box neural networks (regardless of the employed concepts) while remaining globally interpretable. Every possible rule in the memory can be inspected and verified using formal tools, ensuring transparency and trustworthiness. At the same time, the rule selection mechanism ensures that explanations for individual predictions are concise and simple, as only one rule contributed to the final prediction. CMR bridges the gap between interpretability and accuracy, offering a powerful solution for applications that demand both (Figure 3).

If you’re interested in diving deeper into our work, you can read the full paper here or visit our poster at NeurIPS 2024! Our code can be found here.

Please use the following citation if you want to refer to this work:

@inproceedings{debot2024interpretable,
  title={Interpretable Concept-Based Memory Reasoning},
  author={Debot, David and Barbiero, Pietro and Giannini, Francesco and Ciravegna, Gabriele and Diligenti, Michelangelo and Marra, Giuseppe},
  booktitle={Thirty-eight Conference on Neural Information Processing Systems},
  year={2024}
}

References:

[1] Koh, Pang Wei, et al. “Concept bottleneck models.” ICML, 2020.
[2] Barbiero, Pietro, et al. “Interpretable neural-symbolic concept reasoning.” ICML, 2023.
[3] Debot, David, et al. “Interpretable concept-based memory reasoning.” NeurIPS, 2024.