Stop Chunking Code Blindly: Building Better LLM Context with Tree-sitter

Are we giving LLMs too much code when they actually need the right code?

Large Language Models (LLMs) have become remarkably capable at understanding and generating code. However, their ability to reason about a codebase depends heavily on the context we provide them.

For large codebases, providing the entire repository as context is rarely practical. Instead, relevant parts of the codebase need to be identified, organized, and retrieved when required. This makes how we represent and manage code context an important part of building LLM-powered coding systems.

This raises an important question: what if our code chunks could understand that structure?

In this article, we’ll explore how Tree-sitter can help by parsing source code into a structured syntax tree and extracting meaningful code units. We’ll then look at how this structural information can contribute to better context management for LLMs and Code RAG systems.

The Problem with Traditional Code Chunking

When preparing a codebase for an LLM, a common approach is to divide files into fixed-size chunks based on tokens, characters, or lines. While simple, this approach can create several problems.

Consider a simple function:

def calculate_order_total(items, tax_rate):
    subtotal = 0

    for item in items:
        subtotal += item["price"] * item["quantity"]

    tax = subtotal * tax_rate
    total = subtotal + tax

    return total

If we split this file into fixed-size chunks, we might end up with something like:

Chunk 1

def calculate_order_total(items, tax_rate):
  subtotal = 0
  for item in items:
    subtotal += item["price"] * item["quantity"]

Chunk 2

  tax = subtotal * tax_rate
  total = subtotal + tax
return total

Neither chunk represents the complete calculate_order_total function. If a user asks an LLM, “How is the order total calculated?”, retrieving only the second chunk gives the model the tax calculation but loses the logic that produces subtotal.

The core problem is that traditional chunking treats code as plain text, while code has an underlying structure that can be used to build better context for an LLM.

What Is Tree-sitter?

Tree-sitter is a parser generator and incremental parsing library that converts source code into a structured concrete syntax tree (CST). Instead of treating code as a sequence of plain text, it represents its syntactic structure through nodes such as functions, classes, parameters, statements, and expressions. This allows applications to identify and work with meaningful parts of a codebase while preserving their structure, making it useful for building better context for LLMs.

If you’ve used VS Code, you’ve seen how it recognizes functions, variables, and classes and highlights them differently. This kind of code understanding comes from analyzing the structure of the code, rather than treating it as plain text.

Each extracted unit can then be enriched with metadata such as its name, type, file path, line range, parent class, and related imports.

For example, consider this simple Python function:

def calculate_total(price, tax):
  total = price + (price * tax)
  return total

Tree-sitter can represent its structure roughly like this:

FunctionDef
├── name: calculate_total
├── arguments
│ ├── arg: price
│ └── arg: tax
└── body
├── Assign
│ ├── target: total
│ └── BinOp (+)
│ ├── Name: price
│ └── BinOp (*)
│ ├── Name: price
│ └── Name: tax
│
└── Return
└── Name: total

We can identify what each part of the code represents and how those parts are structured. This structural information can later be used to create more meaningful units of context for an LLM.

Get Priyanshi Gupta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Language support:

• Tree-sitter has parsers available for 20+ programming languages through its official ecosystem.
• Supported languages include Python, JavaScript, TypeScript, Java, C, C++, C#, Go, Rust, Ruby, PHP, HTML, CSS, Bash, and others.
• The wider ecosystem also contains many community-maintained grammars, providing support for additional languages.

Turning Extracted Code into LLM Context

Extracting functions, classes, and other code units is only the first step. For an LLM-based context-management system, we also need to preserve information about what the code is, where it comes from, and how it relates to other code.

Instead of storing a function as just a block of text, we can represent it with additional metadata:

{
"type": "function",
"name": "calculate_total",
"file": "utils/orders.py",
"start_line": 10,
"end_line": 15,
"parent": "OrderService",
"imports": ["tax_utils"],
"code": "def calculate_total(price, tax): …"
}

This gives the context manager multiple dimensions of information:

• Code — the actual implementation the LLM may need.
• Metadata — information such as file path, function name, type, and line range.
• Relationships — connections to parent classes, imported modules, called functions, or other related code.

What Happens After Extraction?

Once Tree-sitter has extracted these structured code units, the context-management pipeline can continue with:

1. Enrich — Add metadata and relationships to each code unit.
2. Index — Store the units in a searchable format, using embeddings and/or other retrieval methods.
3. Retrieve — Identify the code units most relevant to the user’s query.
4. Assemble — Combine the selected units and their relevant relationships into a coherent context.
5. Provide — Send the constructed context to the LLM along with the user’s query.

The goal is not simply to retrieve a matching piece of code, but to construct the right context for the LLM at the right time.

Limitations & What’s Missing

Tree-sitter provides a strong structural foundation, but syntax structure alone is not enough to create complete context for an LLM.

Some important information exists beyond the syntax tree:

• Semantic relationships — Which function actually calls another function? Which class depends on which module?
• Runtime behavior — Syntax parsing cannot tell us how code behaves during execution.
• Project-level context — Configuration files, environment variables, documentation, database schemas, and APIs may be important for understanding a code component.
• User intent — The same code may require different context depending on what the user is asking.
• Retrieval quality — Extracting good code units does not guarantee that the correct units will be retrieved.

This means a practical context-management system needs to combine syntax-aware extraction with semantic search, dependency information, documentation, project structure, and query-aware retrieval.

Tree-sitter solves an important part of the problem: it helps us understand what the code is and how it is structured. The next challenge is understanding what context is actually needed for a particular task.

Conclusion

For LLM-powered coding systems, context management is just as important as the model itself. Giving an LLM more code does not necessarily give it better context — the code needs to be structured, filtered, and assembled based on the task.

Tree-sitter provides a useful foundation by allowing us to move from arbitrary text chunks to meaningful, syntax-aware code units. These units can then be enriched with metadata and relationships, indexed for retrieval, and assembled into context when an LLM needs them.

The larger idea is to move from “chunking code for an LLM” to “building context for an LLM.”