OK gpt, explain yourself!

I recently gave a talk on explainable AI to a group of brilliant data scientists. In my talk I used a simple example to demonstrate if ChatGPT can provide a reasonable explanation of its answer. I first asked the audience if they could answer a simple classification problem (with toy data I had created) and explain their answer. Not surprisingly, someone provided the correct answer with a reasonable explanation. I then used ChatGPT to answer the same question and asked for an explanation. Here is how it went down.

Explainability is a crucial aspect of machine learning and artificial intelligence. Explainability is important to ensure safety, detect and reduce bias, build trust and social acceptance in AI, and to debug and audit AI models.

AI models however do not usually provide an explanation of their results. Not only a majority of the AI models are “black boxes” that are not intrinsically explainable, even those that are explainable by nature (such as linear models or decision trees) do not by default provide an explanation — rather, the user must interpret their results based on the knowledge of how the model works.

Image from Christoph Molnar’s Interpretable Machine Learning book (it’s a good book, check it out!)

One might wonder if this could change with the advancements in large language models, such as ChatGPT, which can potentially now explain their results. To see to what extent such models can provide reasonable explanations of their outcome, I gave ChatGPT a simple classification task and asked it to explain its answer.

Get Aras Kayvan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To see how well the gpt model can explain itself, I created a simple toy data set with two classes (red and blue) and a linear decision boundary — see figure below. There are 4 red points above the identity (y = x) line and 3 blue points below it. I gave these labelled points to the gpt model and asked it to predict the colour of the unknown point (marked by a question mark) and, more importantly, explain its answer.

Toy data set with two classes (red and blue) and a linear decision boundary

I actually used this example in a talk on explainable AI and, before showing the audience gpt’s response, I asked if anyone would like to take a stab at labeling this point and explaining their answer. I was giving the talk to room full of brilliant data scientists, so it was no surprise to get the correct answer and a valid explanation: “Blue, because the unknown point is closer to the centre of mass of the blue points than to that of the red points.”

Now let’s see if ChatGPT can solve this simple problem. Here is the prompt and gpt’s response:

ChatGPT has correctly labeled the point. It has also provided an explanation of its response. However, the explanation is not very reasonable and in fact it includes false statements. It goes “This is because all the points with negative y-coordinates are labeled as blue, and point (1,-2) has a negative y-coordinate.” This statement is not false per se, but provides a very weak explanation. It is based on one the features only, and also overlooks the fact that there are two points with y = 0, one red and one blue. It only gets worse from here though. “Additionally, the points with positive x-coordinates are also labeled as blue, but since point (1,-2) does not have a positive x-coordinate, this does not affect our prediction.” Both statements in this sentence are false!

For this experiment I used the gpt-3.5-turbo model. Here is how the get_completion function looks like. If you would like to learn more about this, I recommend watching Andrew Ng and Isa Fulford’s course on deeplearning.ai.

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message["content"]

It will be great if one day computers can explain their results. But it does not seem we are there just yet. Large language models (e.g. the gpt models) may be one possibility toward having explainable results. However, care should be taken in using the output of these models. These models themselves may need to be explained before widespread adoption.