Computation and Language: The Semantic Calculator
Posted by Malkoc Ognjen, AI Researcher of Gaudiy Inc.
In the first part of this series on behavior tree agents, we looked at how behavior trees can bring structure to the often unpredictable world of Large Language Models (LLMs), enabling complex and emergent behaviors. Now, we’re building on that foundation to create something practical: a calculator that understands everyday language. Starting from that base, we’ll explore how the inherent prioritization in behavior trees can be harnessed to interpret natural language and guide meaningful computation.
The core idea put simply; when a task can be solved with explicit instructions in a programming language (rather than relying on the semantic interpretation of an LLM) the outcome is almost always more accurate and reliable. This is especially true for mathematics, where precision is critical.
An illustrative example is arithmetic operations in LLMs. This capability was an emergent property that appeared as models grew larger. More recent work has focused on pinpointing the mechanism: distinct activation circuits seem to support operations like addition or multiplication. This this also points to a limitation — while LLMs can manage basic arithmetic, extending that ability to perform more complex, less common, or out‑of‑distribution operations may be increasingly difficult to learn.
Moreover, reinforcement learning that aligns LLMs by rewarding output containing convincing “reasoning” traces does not appear to improve things. The activation circuits responsible for basic operations seem to remain unchanged, or at least disconnected from the reasoning output⁰. In addition, recent studies ² ³ ⁴ ⁵ further suggest that the benefits of reasoning are not universal. Reference 5, in particular, even links performance gains to specific pre‑training. This implies that if a model is not sufficiently large, or has not been exposed to similar tasks during pre‑training, relying on reasoning alone may be ineffective. It is worth exploring alternatives.
With that in mind, let’s do so. Let’s make the LLM assume the role of a stressed-out doctoral student who does not want to, yet again, derive a symbolic expression, or calculate a well-known, but tedious, integral.
Let’s make use of the collective knowledge we already have at our disposal, tools that have been built specifically for this purpose. In particular, the Python programming language offers several advantages: it has a simple syntax, a vast ecosystem of scientific libraries — and, as luck would have it, there’s also an ever‑growing volume of Python code available online — ample material that feeds directly into the pre‑training phase.
With that back‑story, here’s the plan: we’ll use a behavior tree to enforce the core principle introduced earlier. The rule is straightforward: if a task can be solved programmatically, we solve it that way; for everything else, the LLM steps in as a fallback for solving tasks.
Here we will focus on mathematics and physics. When possible, we hand off numerical work to Python, where well‑chosen function can deliver dependable results. The high‑level idea is what we might call a semantic calculator: for tasks the model itself cannot reliably compute, the LLM acts as a translator between the user and the computer.
The handing off work to Python, and resuming translation by an LLM is trivial since we can leverage the prioritization built into behavior trees. This setup lets us route queries intelligently: whenever a task is computable and expressible in Python, we default to that path, since the results are more dependable. Should that route fail, whether during function generation or result retrieval, we intentionally fall back to the LLM’s natural‑language reasoning. The design ensures a baseline accuracy no worse than using the LLM alone, while delivering clear gains whenever a task can be programmatically computed.
The figure above illustrates how user queries are handled. At the root, the system first checks whether the task can be solved programmatically. If so, the query is routed through the Solve Programmatically sequence, where a conditional node determines whether Python is suitable. If the conditional node return success, the query proceeds to the Get Python Solution node, and the result is passed to the shared Generate Response action.
If the task cannot be solved programmatically, the tree automatically the delegates the task to the LLM as a natural default, which generates a natural‑language response. This design highlights how components like Generate Response can be reused across branches, while the prioritization built into behavior trees ensures that dependable, programmatic solutions are attempted first.
If we peek into the elaborate Get Python Solution node, the explicit plan is,
Now let’s get back on track. We want to use this planning method to improve the accuracy in the output when a user is asking the assistant AI to solve mathematical tasks by leveraging programs to solve problems.
In Practice.
To illustrate the idea, we will instruct the LLM-based assistant, using the three billion parameter instruct-type Llama 3.2 model, to solve the following integral numerically,
We get the Python solution by first looking for a suitable function defined in our tool-belt. For example, we prepare a chunk of text, where we include a set of functions that we can use from within Python to evaluate typical numerical problems. For this example, we include a segment showing what to do to numerically integrate a generic function f,
# Numerical integration of function f, in the interval [a,b]
def integrate(f, a=0, b=1): import scipy.integrate as quad; return quad(f, [a, b])Running this in our semantic calculator, we find that the LLM manages to package our question into an argument that can be used with the tool we prepared. And obtain an explanation of what calculations was done plus the final numerical result.
Already by now, the attentive reader will have some bells ringing; since we’re already working with a generative model, and have repeatedly highlighted Python as a natural fit for code generation in current LLMs, we don’t actually need to spell out the tools when they’re just Python programs. Even smaller LLMs are sufficiently capable to produce valid Python code and handle basic mathematics functions. I’m with the attentive reader on this one — so let’s go ahead and do exactly that.
Fortunately for us, adjusting our planning done by just swapping Find Suitable Function node for Generate Python Code . Running the same query, and leveraging the LLM to 1) interpret the query into arguments for a Python function, and 2) generate the code that outputs the answer we want, we get the following interaction
An extra benefit from using an LLM to generate code, is that it also brings along a layer of common‑sense physics knowledge picked up during pre‑training. To show this in action, we can tackle a simple physics problem that calls for both general domain understanding and a numerical calculation carried out in Python.
For example, let’s study a simple gravitation problem with two solid spheres. This would need knowledge about the gravitational constant, and the simple formula used for calculation the potential energy.
The generated Python script will either define the gravitational constant explicitly or, drawing on Python’s rich scientific ecosystem, retrieve the value directly from a module like scipy. Following our plan, the LLM identifies this as a computable problem that satisfies the Is Suitable for Python condition.
It then produces a pedagogic output that includes:
- a problem statement, clarifying how the query is interpreted,
- an outline of the approach, translating the executed program into a human‑readable explanation, and
- final raw result.
By leveraging the LLM’s ability to bridge natural language and code, it becomes much easier to spot whether the intended approach — or the calculation itself, is faulty. For the example above we get the code,
import numpy as np
from scipy import constants as const
def solve_problem():
"""
Calculates the gravitational potential energy between two masses.
"""
# Constants
G = const.G # Gravitational constant
# Define the positions of the masses
m1 = 4 # mass 1 in kg
m2 = 10 # mass 2 in kg
x1, y1, z1 = 4, 1, 0 # position of mass 1 in meters
x2, y2, z2 = 4, 4, 0 # position of mass 2 in meters
# Calculate the distance between the masses
r = np.sqrt((x2-x1)**2 + (y2-y1)**2 + (z2-z1)**2)
# Calculate the gravitational potential energy
U = -G * m1*m2 / r
return U
# Execute and store result
result = solve_problem()
print("Result:", result)In the background, the generated code is validated to ensure it is truly executable Python. This corresponds to the explicit validation step in the behaviour tree sequence: after the script is produced, it is either executed directly or checked against a set of common errors like non‑ASCII characters, stray markdown markers, or unbalanced parentheses. Once this validation node succeeds, the code is run and the result is formatted so that the final output can be interpreted when generating the response.
The final step in the process is to take that raw number together with the code and translate it into a friendly, human-readable response. It’s this generate response node that turns a simple calculation into a complete answer for the user:
As part of its code generation, the LLM is able to draw on pre‑training in basic physics, allowing it to supply the missing details needed to reach a valid result. Computation is clearly a strength here, but it is only one dimension of what’s possible. The Python ecosystem has accumulated a sea of open tools that extend beyond raw numerical evaluation, tools that also enable symbolic derivations and more advanced forms of analysis.
Get Gaudiy Lab’s stories in your inbox
Join Medium for free to get updates from this writer.
For example, we could use this to solve important questions, like asking what the field would look like from a hypothetical Minovsky particle⁶ field potential of a mobile suit armor,
Here the Python script used to obtain the expression leverages the sympy Python module for symbolic mathematics to compute the partial derivatives, and return the full expression that is used as a result.
import sympy as sp
def solve_problem():
"""
Calculates the gradient of the Minovsky particle field potential V(x,y,z) = kQ/?(x? + y? + z?).
Returns the components of the gradient ?V = (-?V/?x, -?V/?partial y, -?V/?z).
"""
# Define symbols for variables
x, y, z, k, Q = sp.symbols('x y z k Q')
# Define the potential function
V = k * Q / sp.sqrt(x**2 + y**2 + z**2)
# Calculate the partial derivatives
dV_dx = sp.diff(V, x)
dV_dy = sp.diff(V, y)
dV_dz = sp.diff(V, z)
# Compute the gradient
grad_V = (-dV_dx, -dV_dy, -dV_dz)
# Convert the gradient to a list
grad_V_list = [sp.simplify(v) for v in grad_V]
return grad_V_list
result = solve_problem()
print("Result:", result)The key takeaway is that LLM‑driven code generation, together with Python and its ecosystem of tools, provides a unified space for both numerical and symbolic computation, making it possible to translate abstract problems into clear, executable outputs. Crucially, this approach avoids depending on the model to internally “activate the right circuits” for every mathematical operation - a process that is far less controllable, and for which we cannot be certain an LLM will ever achieve complete coverage. By grounding the reasoning in executable code, the results become more reliable, interpretable, and reproducible than those produced by raw language‑model reasoning alone.
But this raises a critical question: how does the system know when to generate code in the first place? The simple approach would be to scan the user’s request for keywords like “calculate” or “solve.” But that’s a brittle solution, language can be too nuanced for simple word matching. What we really want is for the assistant to infer from the broader context of the conversation when a computation is being requested.
Semantic Classification: Orchestra of Models.
At the moment, our system defaults to trying a Python-based solution for any given task. But what if the user is just asking a question or wants to have a conversation? To make our assistant smarter, we need a way to differentiate between a mathematical problem and a more general query. This is where semantic classification comes in. We classify the type of task with a conditional node, where we instruct the LLM to output a JSON containing what class the task has.
This can be difficult for smaller models, and even more so when the task is ambiguous or complex. Fortunately, we can use something more apt for this type of classification — the BERT-class of models. These models are very effective at text classification tasks and can be fine-tuned to recognise specific types of content; in our case, mathematics problems.
We note importantly, this removes the need to author the “perfect” prompt that anticipates the different ways of phrasing. Instead, the model can be fine‑tuned on a set of queries that collectively capture the pattern of interest, even when we ourselves may struggle to articulate it precisely.
Using a IsMathProblem node we decide the branch the LLM should pursue, either a regular language model, or a python generating and executing assistant. We use the classification as a condition by translating the confidence of a predicted class of problems as the criterium.
If we can classify the input as a type of mathematics problem, where any category has a confidence of c>0.85, we will treat it as a computation problem and kick off the Python branch.
By swapping the prompt-based filter with the fine-tuned BERT model, we gain some advantages. It improves our ability to detect a wider range of mathematical queries, eliminates the need to parse an intermediate JSON object, and accomplishes all this with a substantially smaller, more efficient model.
For our proof of concept here, we use a BERT-type model fine-tuned on a dataset of problems from mathematics competitions⁷, where we are interested in specifically which class of problem the user query relates to. The classes are Algebra, Counting & Probability, Geometry, Intermediate Algebra, Number Theory, Prealgebra, and Precalculus. To demonstrate, we ask the LLM to try solving a probability problem involving donuts.
The smaller LLM used in the back-end managed to push through without knowing the analytical solution, by arriving at approximate value via Monte Carlo simulation, solving the problem numerically. (In contrast to a single-shot attempt with Claude Sonnet 4.1, that gave the analytical solution directly)
Now the classifier uses the query to detect which type of problem this is about, and identifies that it with high certainty is about Counting & Probability.
The classification contains a topic whose confidence exceeds the lower-bound for SUCCESS (more than 85%), and the query is processed by generating a Python script.
While the problem is solved with frontier LLMs by providing an analytical formula, the smaller LLM manages to find a brute force approximation using Monte Carlo simulations instead. The code, and accompanying result, is used as a basis for the response presented to the user. The response can be accompanied with the python script that was run, to allow for checking the steps — and it can be explained in a natural flowing conversation.
Now when we input any mathematical problem, we will get a clarification of how the LLM interprets the task, explanation of the approach taken to solve it and the final result; we have reached our goal of making a multi-model, behaviour tree-based semantic calculator!
Concluding remarks
For the sake of brevity, there are a couple of details left out, for example how to use a sandbox that is protected, to execute the python code we generate. Ideally, this allows us to avoid sending messages between frontend and backend if we have stateful behaviour, as we can operate using shared memory.
This proof of concept points to a fundamental shift in how we build intelligent systems. Instead of relying on a single, do-it-all LLM, we demonstrate the power of model orchestration: designing a behaviour plan where a curated set of models work in concert. This flexibility allows us to create an assistant that is not only more accurate and useful but also retains a natural, personalised feel.
Crucially, this approach makes powerful AI more accessible and private. By using smaller, efficient models for specific tasks, we can move away from the data centre and onto the user’s own device. With quantization, these systems can even be loaded into memory and run directly in a web browser, placing control back into the hands of the user.
From here, the path forward branches into exciting possibilities. We can boost performance by integrating cutting-edge code generators — either by leveraging large, closed models like Claude Sonnet and OpenAI’s o4-mini, or by staying unrestricted with dedicated open-source tools like Qwen2.5-Math⁸, or other fine-tuned for code generation⁹. System-wise, we can create more complex, event-driven behaviours that allow disconnected parts of the system to interact. Each path offers a different trade-off, but they all stem from the same core principle: intentional design, where every component is chosen to best serve its purpose.
We will continue to advance empirical research on the application of behavior trees.
References
[1] “Tracing the Thoughts of a Large Language Model” (2025)
https://www.anthropic.com/research/tracing-thoughts-language-model
[2] “Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models.” https://arxiv.org/pdf/2404.15758
[3] “Reasoning Models Can Be Effective Without Thinking.” https://arxiv.org/pdf/2504.09858
[4] “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” https://arxiv.org/pdf/2504.13837
[5] “Spurious Rewards: Rethinking Training Signals in RLVR.” https://arxiv.org/pdf/2506.10947
[6] “Gundam Wiki: Technology — Minovsky Physics.” https://gundam.fandom.com/wiki/Gundam_Wiki:Technology#Minovsky_Physics
[7] Hugging Face model card: bert-finetuned-math-prob-classification. https://huggingface.co/lschlessinger/bert-finetuned-math-prob-classification
[8] Hugging Face model card: Qwen2.5-Math-7B. https://huggingface.co/Qwen/Qwen2.5-Math-7B
[9] Hugging Face model card: CodeLlama-7b-Python-hf. https://huggingface.co/codellama/CodeLlama-7b-Python-hf
More Articles:
About Company:
