Pydantic integration

[miguelgrinberg]

This change adds a few features that support the use of Pydantic models with the DSL module, instead of the standard models defined as subclasses of the AsyncDocument class.

As part of this work some additions have been made to the typing implementation of DSL documents.

• Support for the Annotated syntax when defining document fields in the DSL module. Examples:

class TypedDocAnnotated(AsyncDocument):
    ip: Annotated[Optional[str], field.Ip()]
    k1: Annotated[str, field.Keyword(required=True)]
    k2: Annotated[M[str], field.Keyword()]
    k3: Annotated[str, mapped_field(field.Keyword(), default="foo")]

• Option to exclude a class variable from the list of attributes used to create the ES mapping:

class Doc(AsyncDocument):
    some_var: str = mapped_field(exclude=True)

• New BaseESModel and AsyncBaseESModel classes that inherit from Pydantic's BaseModel and add Elasticsearch superpowers. In particular, any model defined with one of these as its base class will have meta and _doc private attributes and to_doc() and from_doc() methods. The meta attribute includes metadata for each document, things such as id or score. The _doc attribute is a dynamically generated Document or AsyncDocument instance that can be used whenever access to the Elasticsearch index is needed. The methods convert between Pydantic models and ES documents.

  Aside from the extra attributes, this class works exactly like BaseModel and can be used to define data attributes and their validation rules, and the ES document is derived from them automatically. In particular, this class can be used in FastAPI routes, as shown in the quotes example included in this PR. Any annotations intended for the DSL module can be included in the Annotated[] type hint of the respective fields. The Index inner class can be included as well.

class Quote(BaseESModel):
    quote: str
    author: Annotated[str, dsl.Keyword()]
    tags: Annotated[list[str], dsl.Keyword()]
    embedding: Annotated[list[float], dsl.DenseVector()] = Field(init=False, default=[])

    class Index:
        name = 'quotes'

[github-actions]

🔍 Preview links for changed docs

• docs/reference/dsl_how_to_guides.md
• docs/reference/dsl_tutorials.md

[pquentin]

Thanks! LGTM. I only have comments on the example app. I only skimmed the frontend code.

[pquentin]

Isn't this code the same?

Suggested change

	doc = None
	try:
	doc = await Quote._doc.get(id)
	except NotFoundError:
	pass
	if not doc:
	raise HTTPException(status_code=404, detail="Item not found")
	return Quote.from_doc(doc)
	try:
	doc = await Quote._doc.get(id)
	return Quote.from_doc(doc)
	except NotFoundError:
	raise HTTPException(status_code=404, detail="Item not found")

This also applies to get_quote and delete_quote.

[miguelgrinberg]

No, this is close, but not the same. My version doesn't raise any typing errors, yours does. The problem is that our AsyncDocument.get() method can return None if the document is not found and AsyncElasticsearch.get() is not configured to raise the not found exception. That makes all uses of the get() method awkward, because you always have to account for the possible None.

[pquentin]

Why is this needed?

[miguelgrinberg]

This is for security. We do not want the client to decide what the id should be, so by blanking it before calling save() we make sure if they provided one it is discarded.

[pquentin]

Great quotes! Do you maybe have an example that does rely on embeddings?

[miguelgrinberg]

I don't understand what you mean here. This example uses embeddings, both on their own and combined with BM25 search.

[pquentin]

I mean, the results would have been the same with BM25 only

[pquentin]

So maybe there is something else than "dogs and books" that we can use. That's just a nitpick.

[miguelgrinberg]

Ah, I see, you are talking about the screenshot. In fact, "dogs and books" does not match the Groucho Marx quote at the top when using BM25, because that quote has "dog" and "book" in it in singular. I have to test this, but maybe using "canine" instead of dog makes the example more clear and returns the same results.

[miguelgrinberg]

The "books and pets" search term returns a few good ones, including the one from Groucho Marx at the top of the list.