image generated by the author using AI
Yesterday is but today’s memory, and tomorrow is today’s dream. — Khalil Gibran
The mammalian brain can store a large amount of information. This is also possible for LLMs who can safeguard a large amount of knowledge in their parameters. In contrast to LLMs, brains also manage to continuously integrate new experiences without losing previously acquired knowledge.
You Know Nothing, ChatGPT. How Much Does Your LLM Know?
Knowledge is power, but how much an LLM can know, and is it enough?
Get Hey1’s stories in your inbox
Join Medium for free to get updates from this writer.
To try in part to be able to limit this problem, there are solutions such as retrieval-augmented generation (RAG). The RAG is an external memory that allows new information to be presented to the model. LLM then despite the RAG has a static and crystallized memory at the time of pretraining. Thanks to the RAG though, it can retrieve information that is later than the model’s training date.
