What Do Knowledge Graphs and Large Language Models Do to Data? A Perspective on Compression
This topic is one I first began contemplating a year ago. Now, with my Feliz KAG product reaching a milestone, I’m taking the time to document these thoughts.
Last year, I was still being asked whether knowledge graphs held any value after the advent of large language models. This year, a consensus has gradually formed: the solution isn’t an either-or choice but a fusion of both. This is what we now call Graph RAG or KAG (Knowledge-Augmented Generation).
Therefore, this article primarily offers a dialectical interpretation of these two technologies from the perspective of “Knowledge Compression.” Ultimately, I hope to illuminate a clearer path for exploring how they can achieve better synergy and integration.
KG’s “Explicit Compression”
The compression performed by a knowledge graph is a two-stage process: first, the abstraction of domain knowledge into a formal model, the Ontology, and second, the extraction of instances from data sources that conform to this model. The essence of this process is explicit and ontology-driven, with the goal of constructing a highly structured and verifiable representation of factual knowledge.
Stage One: Conceptual Compression. Domain experts and knowledge graph engineers design an Ontology or KG Schema, which serves as a predefined framework. This framework imposes a “forced” simplification on the real world: the continuous spectrum of entities and the infinite subtleties of relationships are refined into a finite, discrete set of classes (e.g., “Person,” “Organization”) and predicates (e.g., “employed by,” “located in”). This modeling act is, in itself, a form of lossy compression driven by business requirements. Any information that does not fit this predefined structure — such as emotion, intent, stylistic nuances, or ambiguity — is discarded. This can be seen as a necessary cost to achieve logical rigor and computational feasibility.
Stage Two: Instance-level Compression. This is what is commonly known as extraction. Using the KG Schema designed in the first stage, automated or semi-automated methods — such as ETL from relational databases, Named Entity Recognition (NER), and Relation Extraction (RE) — are employed to parse unstructured or structured data sources. This allows for the identification and acquisition of instances that match the defined classes and predicates. This process is not a holistic capture of the source data but a highly selective filtering.
For example, a rich and emotive sentence like, “In 1976, a melancholic Steve Jobs, reflecting on his past in his Palo Alto home, co-founded Apple Inc. with his partners,” might be reduced to a few highly simplified triples after extraction, such as: (Steve Jobs, co-founded, Apple Inc.) and (Steve Jobs, residence, Palo Alto). The context, emotion, and narrative style of the original text are lost.
Thus, the final product — the knowledge graph — is an explicit, symbolic representation of knowledge. Its structure is human-readable, its assertions can be independently verified with provenance, and its logical consistency can be checked through formal methods. Its compactness is not derived from encoding latent statistical patterns in the data, but from first abstracting a model of the world and then sparsely populating it with extracted factual instances. It is a compression philosophy that pursues factual “fidelity” over informational “completeness.”
LLM’s “Implicit Compression”
In stark contrast to the KG’s approach, large language models embody an implicit, data-driven compression paradigm. Their compression process does not begin with a predefined schema but is an “emergent property” of their learning objective. This objective is typically singular and deceptively simple: predict the next token in a sequence based on the given context. To continuously optimize for this goal across a massive text corpus, the model is forced to internalize an extremely broad range of patterns into its parameters — from low-level syntax and grammar to high-level semantic logic, factual associations, reasoning patterns, and even the stylistic conventions and societal biases embedded in the training data.
The final product of this compression is not a structured set of facts but the model’s own billions, or even trillions, of parameters (weights), which form a vast and complex high-dimensional vector space. This space can be understood as a Probabilistic World Model. Knowledge is no longer stored as discrete symbols but as intricate models of statistical correlations hidden within all the parameters. An entity like “Steve Jobs” is not a unique node but more like a region or a trajectory in this latent space, defined by its probabilistic relationships with countless other concepts (e.g., “Apple Inc.,” “Pixar,” “minimalism”).
This implicit representation grants the model its extraordinary generative capabilities. When receiving a prompt, an LLM does not “look up” a stored answer. Instead, it samples from a conditional probability distribution based on its internal world model, thereby “computing” in real-time a statistically probable sequence of text as a response. However, this same mechanism is the source of its inherent “lossy” nature. The compression process was not designed for the perfect fidelity of every individual fact but for learning universal, generalizable patterns. Consequently, low-frequency or isolated information may be “averaged out” or incorrectly associated during training, leading to “hallucinations” — the generation of statements that sound plausible but are factually incorrect.
Get Junie Chu’s stories in your inbox
Join Medium for free to get updates from this writer.
In essence, LLM compression sacrifices the absolute verifiability of individual facts in exchange for unparalleled semantic generalization and fluency in text generation. It creates a holistic but approximate model of the world described in its training data.
ChatGPT is a blurry JPEG of the web.
https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web
Fusion and Symbiosis
The profound differences in their compression paradigms do not create an irreconcilable impasse. Instead, a principled, symbiotic fusion between the two is possible.
A prominent example of this fusion is the Retrieval-Augmented Generation (RAG) architecture. In its standard form, RAG enhances an LLM’s generative capabilities by first retrieving relevant text passages from an external knowledge base and then injecting these passages into the prompt as context. This method “anchors” the model’s generation process to specific source materials, mitigating the problem of hallucination to a certain extent. However, when the retrieval source is upgraded from unstructured text to a knowledge graph, the synergy between the two reaches a deeper level. In a “Graph RAG” or “KAG” system, a user’s natural language question is converted into a formal graph query (Feliz AI has a well-designed conversion method for this). What the system retrieves from the KG are not large chunks of unstructured text, but highly structured, unambiguous factual assertions (subgraphs/paths). When these facts are injected into the LLM’s context, its task shifts from “interpreting potentially noisy text passages” to “organizing and narrating a set of pre-validated, structured facts in natural language.”
This symbiotic relationship is bidirectional. LLMs can also act as catalysts, dramatically accelerating the traditionally time-consuming and labor-intensive process of KG Construction. By leveraging the powerful few-shot and even zero-shot learning capabilities of LLMs, efficient entity and relation extraction can be achieved. Similarly, the Feliz AI KAG platform is already capable of automated graph construction.
In this symbiotic loop, the knowledge graph plays the role of the “verifiable source of truth” and “structured reasoning engine,” while the large language model serves as the “fluent natural language interface” and “contextual understander and organizer.” This combination effectively balances the trade-off between verifiability and generalization, creating a hybrid intelligent system that is both knowledgeable and eloquent, precise and reliable, yet flexible and versatile.
Some Conclusions
This article has offered a reconstructive analysis of two major paradigms in contemporary AI — Symbolism (Knowledge Graphs) and Connectionism (Large Language Models) — through the lens of knowledge compression. These two technologies are not superior or inferior to one another; rather, they reflect different design philosophies on the core trade-off between “verifiability” and “generalization.” By acknowledging and understanding this fundamental difference, we can better leverage their respective strengths to achieve effective fusion in practical business applications.
Thanks to fusion architectures like “KAG,” we have designed the Feliz KAG system. It can leverage the determinism of symbolic knowledge to constrain and guide the probabilistic nature of model generation, thereby achieving a higher level of reliability and flexibility within a single system.
Hopefully, on the path toward AGI, various technologies will complement each other, each becoming indispensable to the success of the others.
—
This picture was saved last year, and I feel it pairs perfectly with this article.
