Add annotations next to model completion candidates

[benthamite]

Continuing #315, I have added a commit that implements what we discussed for the Anthropic models. Let me know if this is what you had in mind, and if so I’ll proceed to make the relevant changes for the remaining models.

[benthamite]

Also, given the amount of information now available for each model, perhaps we should allow the user to customize which specific properties are displayed, and/or decide which will be displayed by default.

[karthink]

Thank you, this looks good so far. We need to add similar model information defconsts for Open AI and Gemini models -- it can be done later if you're not interested in adding those as part of this PR.

allow the user to customize which specific properties are displayed, and/or decide which will be displayed by default.

Displayed where? As annotations when selecting a backend/model, or displayed in the mode-line/header line in chat buffers?

[benthamite]

We need to add similar model information
defconsts for Open AI and Gemini models -- it can be done later if you're not
interested in adding those as part of this PR.

I’m happy to add this now.

Displayed where? As annotations when selecting a backend/model, or displayed in
the mode-line/header line in chat buffers?

As annotations.

[benthamite]

I have now finished adding the relevant information for Claude, Anthropic and OpenAI. Note that the gptel code is structured somewhat differently for OpenAI. I used a constant in this case as well, but an alternative is to store this information directly in the variable gptel--openai (rather than in the constant gptel--openai-models). The gptel-make-openai docstring should then be updated to point to that variable or constant for an explanation of the relevant keys, like gptel-make-gemini and gptel-make-anthropic currently do.

[benthamite]

One more thing: I didn’t add information for :capabilities and :mime-types because I haven’t familiarized myself with these properties. If it is clear to you what the values should be in each case, maybe you can add them yourself? Otherwise, let me know and I’ll look into it.

[karthink]

I didn’t add information for :capabilities and :mime-types because I haven’t familiarized myself with these properties.

No worries, we can get to it later (i.e. not as part of this PR).

• :capabilities are input and output types the model supports. The names are made up by me -- media for media input support, json for JSON output, tool for function calling and so on. Other possibilities include image-gen for image output, audio-gen for audio output, embedding for embedding vector output. gptel doesn't have the code to handle any of these except media yet.
• :mime-types are the file MIME types recognized by models with media support.

[karthink]

Looks great so far, thanks! I particularly like that you included links to the sources for this information in the docstrings.

The last remaining step is to update the annotation function in gptel--infix-provider to show these details.

perhaps we should allow the user to customize which specific properties are displayed, and/or decide which will be displayed by default.

I don't think it's worth making it customizable at this stage, since this is only relevant as a discoverability feature when changing models. We can always make the details customizable later.

The screenshot you had in #315 looks like a good set of defaults. To save screen space we could list the the context size as 32k instead of 32000, and truncate the description to ~60 chars or so. The only additional piece of information available is the :capabilities slot. I'll leave the details to you, with one suggestion. I do think including the :capabilities in the annotation is important, since users might want to be able to know when they switch models if the model they pick has support for images or other features.

[benthamite]

Thanks; I just pushed a commit with the updated annotation function. Here’s a screenshot:

A couple of comments:

• The column order, sizes and labels are just what seemed sensible to me after thinking about it for a few minutes. I have very little experience with these types of completion tables, so feel free to suggest changes.
• The input and output cost is displayed in a single column. It may not be immediately obvious to the user what the two numbers represent (though I guess it’s not very hard to infer).
• Relatedly, if the user defines a model for which they supply input or output cost info, but not both, this information will not be shown at all. One could introduce a condition to just display whatever information is available, but this may come at the cost of making the interface less intuitive, since if there is only one number, it is unclear whether this refers to input or to output cost.

The concerns in the two final comments could be addressed by assigning different columns to these two types of cost, though that would make the interface slightly more cluttered.

[benthamite]

Oh, and my revision doesn’t handle cases when a field exceeds its allotted size. I’m not sure what the proper way to handle this is (though I can easily find out with the help of gptel 😉). The two fields for which this is most likely to happen are the model name and its description. I didn’t look into it since I’m not sure you want to introduce more complexity to the code, but I’m happy to do it.

[karthink]

Here’s a screenshot

To save some space I think we can omit the prefixes ("Context: ", "Updated: ", "Cost: "), these are fairly obvious from, uh, context. EDIT: The meaning of the date and cost fields should be obvious from the date and the $ symbol. For the context perhaps we need a prefix?

It may not be immediately obvious to the user what the two numbers represent

For the cost specifically, perhaps something like $2.5 in, $10 out? Just thinking out aloud, we can probably find something better.

my revision doesn’t handle cases when a field exceeds its allotted size

It will just "overflow" and push the remaining columns to the right, I think that's fine.

I suggest changing the ordering slightly to

model description capabilities context cost updated

although any ordering of the last three fields should be fine.

[benthamite]

I just pushed a commit addressing your comments. I tried a few variants and this one is the one I liked the most. We could add a prefix for the context window field, though it looked rather ugly when I tried it, given that none of the other fields did.

[karthink]

1080p screen full width:

1000 px window:

720 px window:

The display breaks down as the window width shrinks, a common problem with annotation strings with relative spacing. I think we need to prepend a space to the description at least.

It might be a good idea to use fixed width fields, truncating the description to the field width when displaying it. This is how other packages like Citar show tabular data -- it remains stable at lower window widths:

[karthink]

Here's what I'm trying right now as the annotation function:

(lambda (comp)
  (let* ((model (nth 2 (assoc comp models-alist)))
         (desc (get model :description))
         (caps (get model :capabilities))
         (context (get model :context-window))
         (input-cost (get model :input-cost))
         (output-cost (get model :output-cost))
         (updated (get model :updated)))
    (when (or desc caps context input-cost output-cost updated)
      (concat
       (propertize " " 'display `(space :align-to 40))
       (when desc (truncate-string-to-width desc 70 nil ? t t))
       " " (propertize " " 'display `(space :align-to 112))
       (when caps (prin1-to-string caps))
       " " (propertize " " 'display `(space :align-to 138))
       (when context (format "%6dk" context))
       " " (propertize " " 'display `(space :align-to 145))
       (when input-cost (format "%10s" (format "$%.2f in" input-cost)))
       " " (propertize " " 'display `(space :align-to 154))
       (when output-cost (format "%11s" (format "$%.2f out" output-cost)))
       " " (propertize " " 'display `(space :align-to 170))
       updated))))

It works better at all window widths, but it's not adaptive -- you need a minimum window width of around 180 chars to see a full annotation string.

[benthamite]

The adaptiveness problem could potentially be solved by setting the width of the fields to either the longest string to be displayed for that field (plus padding) or an upper string length limit, whichever is shorter. In your code, for desc the width would be the minimum of either 70 or the longest model description. I think the only fields that would require this treatment are the model name, :description and :capabilities, which are by far the ones that vary the most in length, though one can extend it to all fields if desired. What do you think?

[karthink]

The adaptiveness problem could potentially be solved by setting the width of the fields to either the longest string to be displayed for that field (plus padding) or an upper string length limit, whichever is shorter. In your code, for desc the width would be the minimum of either 70 or the longest model description. I think the only fields that would require this treatment are the model name, :description and :capabilities, which are by far the ones that vary the most in length, though one can extend it to all fields if desired. What do you think?

• I think the :description column can be fixed at 75 chars or so, see my use of truncate-string-to-width in the previous message, which includes truncation and padding to 75 chars.
• The width of the :capabilities column can be adapted based on the longest one, as you suggest.
• The other columns can all be fixed width, also as you suggest.
• It makes more sense for the input-cost, output-cost and context columns to be right-aligned, with format spec %6d for the context, and so on. I mean something like this:

... (media tool)    16k   $ 0.02 in, $ 0.50 out  2024-04
... (media)       1000k   $30.00 in, $60.00 out  2024-06-14

[benthamite]

The latest commits shorten the :description fields of all OpenAI, Anthropic and Gemini models to make them 70 characters or less (mirroring the default docstring length limit in Elisp) and implement your other suggestions, except for the one concerning the :capabilities field (which I made originally). I think it’s just simpler to truncate that string to 21 characters, which is the minimum length that shows all existing values without truncation. I suggest trying this for now and if this proves too rigid we can refine the implementation later.

[benthamite]

I see that there is one more case that we need to handle: when the length of the model name is greater than the value of :align-to in the desc line (currently 40). Thoughts?

It’s not problematic here because I didn’t add descriptions for the Llama models. However, this is just a contingency of this particular example.

[karthink]

I see that there is one more case that we need to handle: when the length of the model name is greater than the value of `:align-to` in the `desc` line (currently 40). Thoughts?

I think 40 is a reasonable number for backend+model name, but there are a few exceptions. For example, I have TogetherAI:mistralai/Mixtral-8x7B-Instruct-v0.1 which is 48 chars. Setting the width to 50 or the length of the longest string (48 here) for every entry seems like a waste of window space most of the time. Names like this are uncommon, I think we can get away with a field width of 40. What do you think?

[benthamite]

I’d agree with you. The alternatives would be to truncate the model name in the prompt or to set the length of this field dynamically based on the longest model name in the completion candidates. Frankly, I don’t know how to implement either of those options (I spent some time trying to develop something with Claude but all efforts failed).
So if you think what we currently have is acceptable, I’d say let’s go with it.

[karthink]

The latest commits shorten the :description fields of all OpenAI, Anthropic and Gemini models to make them 70 characters or less (mirroring the default docstring length limit in Elisp) and implement your other suggestions, except for the one concerning the :capabilities field (which I made originally). I think it’s just simpler to truncate that string to 21 characters, which is the minimum length that shows all existing values without truncation. I suggest trying this for now and if this proves too rigid we can refine the implementation later.

Works for me.

It looks like the ellipses character in truncated strings is pushing things out by one char, making the display misaligned:

I've solved this in the past but can't remember how I did it now. 😞

[benthamite]

Strange, for some reason it’s not happening for me.

(I just added some random words to the original docstring; hence the nonsensical final word.)

[karthink]

Yeah, your ellipses character is single width, while mine isn't, or at least mine isn't as narrow. It depends on the font and some other details I don't understand. You can check by running (truncate-string-ellipsis).

EDIT: Just checked with a few other monospace fonts -- they work fine. Only Iosevka seems to have this problem.

[karthink]

I think we've wandered pretty far from displaying model information into discussing the minutiae of char widths in different fonts. gptel, like Emacs, is always a little broken. I think what you have right now is fine!

I had a final few comments:

• When there is no description or capabilities, the remaining fields appear to be misaligned (see highlighted entry here):
  

• Since the input-cost and output-cost are closely related, I suggest using a comma between them, with one space instead of two (other fields are separated by two spaces at full width)

• I suggest aligning the dollar signs and the decimal points in the cost fields.

The above two suggestions are demo'd in this example:

... (media tool)    16k   $ 0.02 in, $ 0.50 out  2024-04
... (media)       1000k   $30.00 in, $60.00 out  2024-06-14

• I think the models gpt-4 and gpt-4o now point to models updated in 2024? The updated-date still says 2023.

[benthamite]

I’ll work on implementing your suggestions. In the meantime, take a look at the screenshot, which shows that the first of the issues you mention does not happen on my end (unless I’m missing something).

[karthink]

I’ll work on implementing your suggestions. In the meantime, take a look at the screenshot, which shows that the first of the issues you mention does not happen on my end (unless I’m missing something).

Hmm, I'm not sure. I'll wait for the final version and test again.

[benthamite]

I have now processed your suggestions.

When there is no description or capabilities, the remaining fields appear to be misaligned

Not done, since I was not able to reproduce the issue (as mentioned).

Since the input-cost and output-cost are closely related, I suggest using a comma between them, with one space instead of two (other fields are separated by two spaces at full width)

Done.

I suggest aligning the dollar signs and the decimal points in the cost fields.

Done.

I think the models gpt-4 and gpt-4o now point to models updated in 2024? The updated-date still says 2023.

Not done, since I was not able to confirm this. The official GPT-4 page still lists October 2023. Furthermore, when I asked GPT-4o, it responded that the cutoff dates, for both GPT-4 and GPT-4o, are October 2023.

Separately, I have replaced :updated with :cutoff-date, since the latter appears to be the term commonly used to refer to this, and :updated is a bit vague, since it could refer to other types of update (e.g. the date the information in the annotation was last updated). If you don’t like this, we can always revert the commit.

[benthamite]

I fixed a few more things. I’m pretty happy with the current version, so unless you have further suggestions, I think it can be merged.

[karthink]

Looks good! Thanks for your hard work @benthamite.

Here's hoping Emacs gets an easier way to display tabular data in annotation functions, the way tabulated-list-mode does in buffers.

[benthamite]

Great! I’m happy to have contributed to what is my all-time favorite Emacs package.