From fancy bubble tea to practical business automation — exploring two approaches to making AI genuinely useful.

Introduction: AI, we need to talk. Or maybe not?

Recently, I came across an interesting development in AI that made me wonder: do we really need a Large Language Model (LLM) to answer every question?

Think about it. We have powerful AI models capable of writing articles, generating code, explaining quantum physics, and occasionally telling us how wonderfully intelligent our questions are.

But what if our application needs something much simpler?

Should we approve this order? Is this customer likely to leave? Which product should we recommend?

We don’t necessarily need a three-paragraph explanation from an AI philosopher. We need a decision, preferably fast, reliable, and understandable by software.

This is where two interesting AI approaches come into play: Jev, a new decision-focused model from TypeSafe AI, and Tabular Foundation Models, such as TabPFN and SAP’s RPT.

They approach the same problem from different directions. And, interestingly enough, they might work even better together.

1. Jev: Less talking, more deciding

Jev is the first System One model from TypeSafe AI. Instead of generating sentences word by word, it produces structured decisions that ordinary application code can use directly.

The name System One comes from Daniel Kahneman’s concept of fast, intuitive thinking.

Imagine an online shop selling some rather unusual products:

• Unicorn Bubble Tea
• Dragon’s Breath Chili
• Fairy Dust Tiramisu
• Wizard’s Cloak Hoodie

A customer writes:

“I love spicy food, but last time your chili almost turned me into a fire-breathing dragon. Could you recommend something less dangerous?”

A traditional LLM might compose a lovely response with apologies, recommendations, and perhaps a joke.

Jev takes another approach. We pass the customer’s message as state (the information available at the moment) and ask specific questions.

For example, Jev could classify the customer’s request as a product recommendation rather than a refund request. It could also return probabilities for the available choices.

No essay. No unnecessary conversation. Just a structured decision that application code can use.

Under the hood, Jev uses a training approach called Reinforcement Learning for Calibrated Decisions (RLCD). The idea is to reward not merely choosing the right answer, but also providing useful estimates of uncertainty.

It can produce a choice, a yes/no probability, or a score on a predefined scale. Multiple questions can be answered in parallel.

TypeSafe reports response times of 70–500 milliseconds for its target workloads, although actual performance depends on the task and deployment.

One important detail: Jev cannot generate arbitrary prose, but that does not mean its decisions are guaranteed to be correct. A perfectly valid answer can still be wrong.

2. Tabular AI: The spreadsheet gets a brain

Now consider a different situation.

Our imaginary shop has been selling magical products for several months. Naturally, we’ve collected some data.

Nothing particularly fancy. Just rows, columns, and numerical values. This is what we call tabular data.

Traditional machine learning has been working with this type of data for decades, using algorithms such as Random Forest, XGBoost, and CatBoost.

But a newer generation of models, including TabPFN and SAP’s Relational Pretrained Transformer (RPT), offers an interesting twist.

Instead of learning everything from scratch for each new spreadsheet, these models arrive with knowledge gained through earlier training on many datasets.

Think of it like hiring a data analyst who has already seen millions of spreadsheets.

You provide examples of previous sales, including which products were successful. The model looks for relationships between the columns and predicts outcomes for new records.

For example:

“Will Fairy Dust Tiramisu sell well during our Halloween campaign?”

Tabular AI could estimate demand using relevant historical records, prices, promotions, seasons, and other features.

The interesting term here is in-context learning: certain tabular foundation models can use the labeled examples supplied with a prediction request without traditional task-specific retraining.

TabPFN learns much of this ability during pretraining on synthetic datasets. Its 2025 research demonstrated strong performance on relatively small tabular datasets. The family has evolved quickly since then: TabPFN-2.5 extended support to datasets of up to 50,000 records and 2,000 features, while the newer TabPFN-3.5 continues pushing the capabilities of tabular foundation models.

And TabPFN isn’t alone in this space.

NVIDIA recently introduced Kumo Tabular, another foundation model that learns from labeled rows and predicts new outcomes without traditional task-specific retraining. NVIDIA reports impressive accuracy and efficiency across several tabular benchmarks.

SAP's Relational Pretrained Transformer (RPT) takes a related approach. NVIDIA's Kumo Relational also works with relationships across multiple database tables, moving beyond a single spreadsheet.

Different engines, similar ambition: turning ordinary business data into useful predictions without building a new machine-learning pipeline for every question.

Of course, a model can’t discover demand signals that were never recorded. If no relevant seasonal information exists, Halloween remains a mystery — even for a sophisticated transformer.

3. Jev vs. Tabular AI: Same destination, different engines

Here is where things become interesting.

Both technologies are useful for making predictions, assigning scores, and supporting decisions. But they rely on different kinds of information.

Zero-shot vs. few-shot deserves a simple explanation.

Imagine asking two chefs to prepare Dragon’s Breath Chili.

The first chef has general cooking knowledge and can make a judgment immediately, without tasting previous batches. That’s our Jev analogy.

The second chef has records of 500 previous recipes, ingredients, customer ratings, and complaints about excessive spiciness. They use these examples to estimate whether the next recipe will succeed. That’s tabular AI.

Neither chef knows everything. One relies more on general knowledge; the other learns from your specific historical examples.

And this is precisely why combining them can make sense.

4. What if we combine both? The magical shop experiment

Suppose our shop wants to predict whether a customer will return a product.

We have structured information, such as purchase history and refund rates, alongside free-form customer reviews.

Jev can interpret the text:

“The Wizard’s Cloak Hoodie looks fantastic, but the sleeves are designed for an actual wizard with six arms.”

It might classify the review as a sizing complaint with a high probability.

Tabular AI can then combine that judgment with historical return patterns.

We could use the resulting prediction to offer a size exchange, recommend another product, or ask a human to review an unusual situation.

And this isn’t purely hypothetical from an architectural perspective. SAP researchers have already explored treating tabular foundation models as decision models and comparing their RPT approach against Jev.

In an SAP-reported comparison across four enterprise workflows, Jev achieved 72.70% accuracy, SAP RPT reached 76.75%, and RPT combined with CASE achieved 78.75%.

The additional CASE component helps RPT interpret semantic information in text.

One important detail: Jev was evaluated zero-shot, making decisions without task-specific labeled examples. SAP RPT used a many-shot approach, drawing on a larger collection of labeled examples.

Both approaches have their strengths, but the different learning setups mean this isn’t a like-for-like leaderboard.

5. What about real-world implementation?

If I were building a proof of concept, I would start with something deliberately simple.

Take a CSV file containing order history, add a collection of customer messages, and define one prediction target: Will this order require manual review?

For the first prototype:

1. Use Jev’s API to classify customer messages into predefined categories, with probabilities.
2. Add those judgments as new columns to the dataset.
3. Feed the enriched table into TabPFN or another suitable tabular model.
4. Test predictions against previously unseen orders, and compare with Jev alone and a traditional CatBoost baseline.

The important part is evaluating on data the system hasn’t already learned from. Otherwise, impressive-looking accuracy could simply mean we’ve built a very expensive memorization machine.

In production, we would also need to check probability calibration, false positives, privacy requirements, inference costs, and what happens when the model isn’t confident enough.

For practical experimentation, the starting points are TypeSafe AI and Prior Labs TabPFN.

6. So, is this the future of AI?

Possibly one part of it. But let’s not throw away our LLMs just yet.

Jev represents an interesting move toward AI as a software component rather than an endlessly chatting assistant. Tabular foundation models bring a different advantage: using previously observed relationships to predict what might happen next.

There are still limitations. Jev’s internal architecture is not fully public, and its calibrated probabilities need real-world validation. An independent September 2026 evaluation found strong results on several classification tasks, but also weaknesses in fine-grained judgments and certain probability thresholds.

Meanwhile, tabular models remain sensitive to data quality, changing business conditions, and the examples we feed them.

The fundamental lesson is surprisingly simple.

Not every AI problem needs a language model. Not every prediction needs a chatbot. And not every business decision requires a giant neural network.

Sometimes you need an LLM to explain a complicated problem. Sometimes you need Jev to make a quick judgment. Sometimes you need a tabular model to discover what your business data has been trying to tell you for years.

And sometimes you need all three.

And if anyone manages to predict the optimal spiciness of Dragon’s Breath Chili without setting the kitchen on fire, I’d be interested in the dataset.

Further reading

• TypeSafe AI — Introducing Jev (September 2026)
• Prior Labs — TabPFN technical reports
• TabPFN-3.5 Technical Report
• NVIDIA — Kumo Tabular
• SAP — Jev vs. RPT
• Evaluating and Benchmarking the System One Model Jev

Recently, I was inspired by a video on the NVIDIA AI channel, showcasing the Gemma 2 language model from DeepMind, which has 2 billion parameters, running on a Jetson Orin Nano. Impressive stuff! This spurred me to experiment with my Raspberry Pi 5, where I chose to run the Phi-3 mini 4K Instruct, a 3.8 billion parameter LLM from Microsoft, using the ONNX Runtime GenAI framework. Despite its considerable size, this model is surprisingly nimble, delivering about 4 tokens per second. Moreover, this same setup is also capable of running Mistral 7B!

Quick video demonstration of how the Phi-3 mini 4K Instruct LLM performs real-time CPU inference on a Raspberry Pi 5, with the script executed on the device via SSH console:

Impressive, isn’t it? This is just one step closer to a future integrated with AI. The entire setup was configured using compiled ONNX Runtime libraries and a straightforward Python script, along with the open-source Phi-3 LLM, which I downloaded directly onto the Raspberry Pi 5 running the latest version of Ubuntu OS. I’m excited to share a detailed step-by-step guide in this article, along with some intriguing insights about the technology and the hardware involved.

Interesting insight: What we currently define as Large Language Models (LLMs) could be considered Small Language Models (SLMs) in future contexts. As technology progresses, what seems vast today might become the standard or even minimal tomorrow, illustrating the rapid pace of advancement in AI model development. This fluidity challenges our understanding and categorization of language models, constantly pushing the boundaries of what is considered large or small.

Although I tested this setup on Ubuntu, the instructions provided should also be applicable to Raspberry Pi OS. This guide aims to give you a clear idea of what’s possible with AI on less conventional hardware platforms and how to leverage these systems for powerful, efficient computing.

ONNX Runtime / ONNX Runtime GenAI

ONNX Runtime is a performance-built engine designed to execute machine learning models serialized in the ONNX (Open Neural Network Exchange) format. It is compatible with models trained on a variety of frameworks, including PyTorch and TensorFlow, which enhances its adaptability across different platforms.

Capable of utilizing both CPUs and GPUs, along with other hardware accelerators, ONNX Runtime’s flexibility makes it suitable for diverse computing environments — from personal computers to high-end servers.

The ONNX Runtime GenAI package is a refined version of ONNX Runtime, specifically optimized for generative AI models. It enhances performance for tasks like image generation and natural language processing by integrating specific hardware and software optimizations.

This guide will walk you through setting up the ONNX Runtime GenAI package from source on a Raspberry Pi 5, leveraging Ubuntu and the powerful Arm64 CPU for inference tasks.

Raspberry Pi 5 Hardware Overview

The Raspberry Pi 5 is equipped with a quad-core Arm Cortex-A76 processor and a 12-core VideoCore VII 800MHz GPU. The Cortex-A76 operates on the ARMv8-A architecture — a robust 64-bit platform also known as ‘AArch64.’ Renowned for its strong performance, this architecture is utilized across a spectrum of devices, from smartphones to sophisticated computing systems.

Python and Virtual Environment

Setting up a virtual environment is optional if you are already familiar with Python environments. Virtual environments are essential for project context switching, allowing you to isolate software dependencies across different projects on the same machine.

Step 1: Verify Python Installation

Start by checking if Python is installed on your system:

python3 -V

You should see output similar to Python 3.12.3.

Step 2: Create and Activate a Virtual Environment

Create a new virtual environment in your preferred directory:

python3 -m venv onnxruntime-genai-env

Activate it:

source onnxruntime-genai-env/bin/activate

Install the Virtual Environment Tool (if necessary)

If your system does not already have the virtual environment tool installed, set it up with the following command:

sudo apt-get install -y python3-venv

Step 3: Update pip

Ensure that your pip installation is up to date to avoid any compatibility issues:

pip install --upgrade pip

By following these steps, you will establish a dedicated environment tailored for working with the ONNX Runtime GenAI package, effectively isolating it from other projects and dependencies.

Install Build and Development Tools

Before you begin, ensure that your system has the necessary building packages installed. These packages, which include compilers and libraries essential for compiling software from source, may not have been previously installed on your system. Install them with the following command:

sudo apt-get install -y build-essential cmake

Next, install the development utilities needed for this guide. These tools are crucial for downloading sources and managing code:

sudo apt-get install -y curl wget tar gzip git

Executing these commands equips your system with the foundational tools required for building and managing the ONNX Runtime GenAI package. This setup ensures all prerequisites are met for a successful installation. These tools will also be invaluable for various future projects, not just this build.

Get ONNX Runtime GenAI

1. Clone the Repository

Start by cloning the ONNX Runtime GenAI project files from the official GitHub repository:

git clone https://github.com/microsoft/onnxruntime-genai.git onnxruntime-genai

Once cloned, navigate to the onnxruntime-genai directory:

cd onnxruntime-genai

2. Checkout the Specific Version

To work with the specific version required for your project, switch to version v0.3.0:

git checkout tags/v0.3.0

Note: At the time of writing, higher versions of ONNX Runtime GenAI require the nightly build of ONNX Runtime. This requirement adds complexity to the setup process. While still manageable, it may necessitate additional steps such as installing packages via NuGet or downloading pre-compiled versions from the Visual Studio servers.

3. Verify the Version

Ensure you have checked out the correct version by running:

git status

The output should confirm HEAD detached at v0.3.0.

Get ONNX Runtime

The ONNX Runtime is required to build the GenAI package. Follow these steps to download and set it up correctly. For more information, visit the ONNX Runtime GitHub releases page to access the latest version compatible with your architecture.

1. Fetch the Download Link

To obtain the download link for the version that matches your system, use the following command:

curl -s https://api.github.com/repos/microsoft/onnxruntime/releases | grep browser_download_url | grep 'aarch64' | head -n 1 | cut -d '"' -f 4

For instance, this command might output a URL such as:

• https://github.com/microsoft/onnxruntime/releases/download/v1.19.2/onnxruntime-linux-aarch64-1.19.2.tgz

This version has been tested and confirmed to work, hence it is recommended to use it to avoid compatibility issues.

2. Download the Release

Download the specified version using:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.2/onnxruntime-linux-aarch64-1.19.2.tgz

3. Extract and Setup the ONNX Runtime Files

Once downloaded, you need to extract the files and set them up within your onnxruntime-genai directory:

tar -xzvf onnxruntime-linux-aarch64–1.19.2.tgz
mv onnxruntime-linux-aarch64–1.19.2 ort

Make sure these commands are executed within the onnxruntime-genai directory to ensure correct pathing. The proper setup of the lib and include directories is crucial for the building process. Incorrect placement can result in build errors indicating that the ONNX Runtime library could not be found.

Building ONNX Runtime GenAI

Navigate to the onnxruntime-genai directory and begin the build process by executing:

sh build.sh --build_dir=build/Linux --config=RelWithDebInfo

Once the build completes, you will find the Python wheel package in the directory build/Linux/RelWithDebInfo/wheel. Further instructions will be provided in the next step.

Building may take some time, and progress will be displayed, giving you an estimate of the time remaining.

Handling potential issues (skip if no errors)

During the build, especially with pybind11, you might encounter errors related to missing Python headers such as: Imorted target “target” “pybind11::module” includes non-existent path or similar messages. This typically occurs if the system Python installation paths do not match the expected locations by the build scripts.

To resolve this, install the Python Development Version. Using the python-config tool will help specify the correct include directories needed for the build:

sudo apt-get install python3-dev
python3-config --includes

The command above should help resolve the issue and output the paths to the Python headers, which are needed for the build process. The output will look something like this:

-I/usr/include/python3.12 -I/usr/include/python3.12

Install the Built Package

Now that the package is built, it’s ready for installation. For CPU inference, you’ll first need to install NumPy:

pip3 install numpy

Next, install the ONNX Runtime GenAI wheel package from the build directory:

pip3 install build/Linux/RelWithDebInfo/wheel/onnxruntime_genai-0.3.0-cp312-cp312-linux_aarch64.whl

To verify if the installation was successful, use the following Python command:

python3 -c 'import onnxruntime_genai; print(onnxruntime_genai.Model.device_type)'

This command will output the device type that the ONNX Runtime GenAI model is using, which typically appears as:

<property object at 0xfff123abc456>

The specific output isn’t critical; what matters is that the onnxruntime_genai package has been successfully imported and is invokable.

Download Phi-3 ONNX Model

To ensure efficient inference from the outset, we’ll download the Phi-3 mini 4K instruct model in the ONNX format. This model is particularly suited for CPU inference and is optimized with CPU INT4 RTN block 32 and accuracy level 4, aligning perfectly with the requirements of this guide.

Model Background:

• Model Type: Phi-3-mini is a Transformer-based language model.
• Parameter Count: 3.8 billion.
• Training Data: High-quality, educationally valuable datasets, including NLP synthetic texts and chat data from both internal and external sources.
• Post-Training Optimization: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) for enhanced chatting capabilities, alignment, robustness, and safety.

Model Availability:

The model files are hosted on the Hugging Face portal (Phi-3 mini 4K Instruct ONNX).

We will specifically download the CPU-optimized files available here: Phi-3 CPU Files.

Method 1: huggingface-cli (Recommended method)

Using huggingface-cli for simplicity and completeness.

Install huggingface-cli:

pip install -U "huggingface_hub[cli]"

Download only the necessary data:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/* --local-dir ./Phi-3-mini-4k-instruct-onnx
# Download the license file
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include LICENSE --local-dir ./Phi-3-mini-4k-instruct-onnx

Alternatively, to download the entire repository:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx

Method 2: Git clone with LFS (Alternative method)

Install and configure Git Large File Storage (Git LFS) to handle large files:

sudo apt-get install git-lfs
git lfs install
# or alternatively, install via script
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash

Clone the repository:

git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx

Method 3: wget (Alternative method)

This method is less efficient but still available. It requires downloading each file individually. For example, it may look like this:

wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/blob/main/README.md

Mistral 7B ONNX Model (Optional)

For those interested in alternative options, the Mistral 7B model serves as a viable substitute for the Phi-3 model. This section will guide you on how to download and integrate the Mistral 7B model into your Raspberry Pi setup, with specific details on adjusting parameters in the inference script provided later in this guide.

Model Details:

• Optimization: The Mistral 7B Instruct v0.2 is in ONNX format, optimized for CPU performance with an INT4 RTN block 32, achieving accuracy level 4 for efficient CPU inference.
• Location: Explore the Mistral 7B model files on the Hugging Face portal at Mistral 7B model files and Mistral-7B CPU Files for the CPU-specific package.

Downloading the Model:

huggingface-cli download microsoft/mistral-7b-instruct-v0.2-ONNX --include onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4/* --local-dir ./mistral-7b-instruct-v0.2-ONNX

Performance: While the Mistral 7B ONNX model can be used interchangeably with the Phi-3 mini 4K model, it’s important to note that the Mistral 7B may exhibit slower performance compared to the Phi-3 mini 4K. The architectural differences in the Mistral model could impact real-time processing speeds and overall inference performance on the Raspberry Pi 5.

Inference

Included below is a simple script named onnxruntime-genai-test.py designed for streamlined chat conversations. This script encapsulates the minimum required steps for efficient inference.

Create file and Edit the script

1. Create the script file: touch onnxruntime-genai-test.py
2. Open the script: Open onnxruntime-genai-test.py with your preferred text editor.
3. Add the following content: Ensure the model path within the script accurately reflects where the model files were downloaded, adjusting as necessary for your setup.

# Compiled library
import onnxruntime_genai as engine

# Phi-3 mini 4K instruct model path
model_path = "Phi-3-mini-4k-instruct-onnx/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4"
# Mistral 7B instruct v0.2 model path
# model_path = "mistral-7b-instruct-v0.2-ONNX/onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4"

print(f"Loading {model_path} model\n")

model = engine.Model(f'{model_path}')
tokenizer = engine.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()

# Set the maximum number of tokens to generate
# The full example includes additional parameters for more complex configurations
search_options = {'max_length': 2048}

# Template used to format multi-turn conversations in the chat interface
chat_tpl = '<|user|>\n{input}<|end|>\n<|assistant|>'

print("Let's chat!\n")

# Multi-turn chat loop. Exit with Ctrl+C
while True:
    text = input("> User: ")
    if not text:
        print("Please, answer something")
        continue

    # Populate the chat template with user input
    prompt = f'{chat_tpl.format(input=text)}'

    input_tokens = tokenizer.encode(prompt)

    gen_params = engine.GeneratorParams(model)
    gen_params.set_search_options(**search_options)
    gen_params.input_ids = input_tokens
    generator = engine.Generator(model, gen_params)

    print("\n> Assistant: ", end='', flush=True)

    try:
        # Loop to generate and display answer tokens
        while not generator.is_done():
            generator.compute_logits()
            generator.generate_next_token()
            next_token = generator.get_next_tokens()[0]
            print(tokenizer_stream.decode(next_token), end='', flush=True)
        print('\n')
    except KeyboardInterrupt:
        print("\nCtrl+C pressed, break\n")

# Delete the generator
del generator

Script Details

This script implements a basic one-question-one-answer chat system, designed to showcase local AI capabilities using the ONNX Runtime GenAI framework.

This script is adapted from an example on the ONNX Runtime GenAI GitHub repository’s examples page. For more detailed information and guidance on adjusting inference parameters, refer to the original example.

Running the Script

To execute the script, run:

python3 onnxruntime-genai-test.py

Please be aware that the cold start may take some time as the LLM needs to initialize. This process involves loading the model, tokenizer, and necessary data into memory before inference can begin.

To stop or exit the script, use the Ctrl+C keyboard combination.

Enhancing Chat Features

The example of code above operates using the simple prompt template:

<|user|>\n{input}<|end|>\n<|assistant|>\n

For example:

<|user|>
Create a short poem about the ocean.<|end|>
<|assistant|>

Here’s a breakdown of the components:

• <|user|> and <|assistant|>: These tokens denote the roles within the conversation. <|user|> indicates that the text following it is the user's input, while <|assistant|> signals the beginning of the AI's response. This role labeling helps the model recognize who is speaking and adjust the responses accordingly.
• {input} and <|end|>: The {input} placeholder is where the actual user query or command is inserted. The <|end|> token signifies the end of the user's input, which helps the model understand where the input stops and where a response should begin.

The <|...|> tokens are specialized markers used to structure and segment text data during AI model training and in dialogue handling applications. These tokens assist the AI in differentiating between various segments of the dialogue, such as pinpointing when a user starts and stops speaking, or indicating when the AI should commence its response. This convention is specifically tailored to simplify the processing and interpretation of conversations within the model, facilitating easier management and enhancement of dialogue flows.

Multi-turn Conversation

For extended conversations that maintain context and memory, consider modifying the script based on the multi-turn conversation template:

<|user|>\n{input}<|end|>\n<|assistant|>\n{output}<|end|>\n

For example:

<|user|>
What's the future of artificial intelligence?<|end|>
<|assistant|>
The future of artificial intelligence is poised to be transformative across many sectors, including healthcare, transportation, and education. As AI continues to evolve, it is expected to become more integrated into daily life, enhancing personal and professional tasks through automation and advanced analytics.<|end|>

The main difference from the template above is the inclusion of the {output} token and data:

• {output}: This is where the AI's response or output is inserted, along with the <|end|> token. It represents what the AI has generated in response to the user's input, providing information, answering questions, or continuing the conversation.

If you’re interested in adapting the script to handle more complex interactions or maintain a chat history, let’s discuss further in the comments below.

Conclusion

Thank you for reading! Your feedback and support are crucial to authors like myself. If you found this guide helpful, please give it a clap and consider subscribing for more content like this.

This guide completes our journey through setting up on-device inference with the Raspberry Pi 5. If you’re interested in more projects with Raspberry Pi, check out my other guides:

• Easy Guide to Text-to-Speech on Raspberry Pi 5 Using Piper TTS

If you’re using different platforms, you might also find my guide for ARM-based Macs useful:

• Running Phi-3/Mistral 7B LLMs on a Silicon Mac locally: A Step-by-Step Guide

Let’s stay connected! For more insights and ongoing discussions, follow me on:

• LinkedIn
• Twitter

Your support makes these resources possible and fuels future content. Thanks again for your time and engagement!

Piper TTS is a specialized local neural text-to-speech system, optimized for the Raspberry Pi 4 and adaptable to the Raspberry Pi 5. It delivers reliable voice output quickly and can operate offline, within your home network, or online. This versatility makes it ideal for Raspberry Pi projects that lack consistent internet access or are part of a broader, home-based IoT system, such as a specialized lightweight text-to-speech processing server like the one I created for my learning needs.

What is Text-to-Speech?

Text-to-Speech (TTS) technology transforms written text into spoken words, turning your favorite articles into narrators and books into audio-books. But how does this happen? Let’s delve into the mechanics behind the magic.

Integrating Piper TTS into your setup transforms the device into an interactive component of your technology ecosystem, enhancing its functionality and broadening its range of applications. Essentially, you’re giving a voice to your Raspberry Pi, making it a central player in your tech setup.

The real caviar here is that the Raspberry Pi runs Python scripts, which makes it programmable in virtually any way you can imagine. This allows for extensive customization and adaptability in your projects, from simple automated tasks to complex IoT systems.

Why Opt for Raspberry Pi in IoT Projects?

Choosing a Raspberry Pi — or something similar — over conventional devices like laptops, PCs, or tablets is often driven by the device’s suitability for permanent, IoT-focused installations. The Raspberry Pi combines cost-effectiveness with flexibility and sufficient processing power, accommodating a diverse array of applications from digital assistants to home media systems and environmental monitoring stations. These devices are designed to remain operational continuously, making them ideal for long-term, data-intensive tasks.

A Bit of Spice on the Tech Front

According to various reviews, the Raspberry Pi 5 is declared to be about 2–3 times faster than its predecessor in terms of compute power. It boasts faster I/O speeds and more memory in the top model. While each specific task requires separate benchmarking to obtain precise numbers, these estimates are helpful for broad comparisons.

For tasks such as these, I prefer using Docker. It offers a scalable and isolated solution that can be wrapped in a convenient API and easily documented. In this article, I’ll employ the same approach using Ubuntu for ARM64 as the base system. These instructions should also be transferable to setups using Raspberry Pi OS, with the primary requirement being that the Docker environment is correctly set up.

You can find comprehensive details about Piper TTS on its GitHub page. For step-by-step instructions on how to build from the source or utilize the precompiled binary package, please refer to the Piper TTS Installation Guide. While there is no ready-made recipe specifically for Raspberry Pi 5, the binary built for AArch64, available at piper_linux_aarch64.tar.gz, functions well. However, we will proceed with building the package from source to provide a detailed explanation of the process.

Note: For Raspberry Pi 4, the piper installation can be done simply with the command pip install piper-tts, rather than building it from source.

Piper Voices

Ready-to-use voice packages for Piper are available on Hugging Face and in the VOICE.md file on the Piper GitHub page. Each package includes two essential files: the model itself in ONNX format and the JSON config file. Additionally, the MODEL_CARD provides licensing information and detailed descriptions about each model and its usage. The example model aru medium is included in the following Docker package, but you may choose to download more voices according to your preferences.

Installing Docker Engine on Ubuntu

Ensure Docker Engine is properly installed on your Ubuntu system by following these steps. If Docker is already installed on your system, you can skip this section and proceed to the Dockerfile chapter. For a more comprehensive guide, refer to the official Docker installation guide for Ubuntu.

Remove potentially conflicting packages

for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

Add Docker’s official GPG key

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the repository

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Update Packages

sudo apt-get update

Install latest packages

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Verify Installation

sudo docker run hello-world

Dockerfile

Create a Dockerfile in your working directory with the following content:

# Use a specific, stable version of Ubuntu
FROM ubuntu:24.04 AS main

ARG DEBIAN_FRONTEND=noninteractive

# Replace default shell with bash
RUN rm /bin/sh && ln -s /bin/bash /bin/sh

# Set the path for the code directory
ENV CODE_PATH=/opt

# Update and install necessary packages in one layer, clean up in the same layer
RUN apt-get update && DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata \
    build-essential apt-utils vim bzip2 tar gzip git curl wget python3 python3-pip cmake ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Set up Python pip configuration
RUN echo $'[global]\nbreak-system-packages = true' > /etc/pip.conf

# Upgrade pip
RUN pip install --quiet --upgrade pip --ignore-installed

# Clone the Piper TTS repository
RUN git clone https://github.com/rhasspy/piper.git $CODE_PATH/piper
WORKDIR $CODE_PATH/piper
# Use a tested version of Piper
RUN git checkout tags/2023.11.14-2

# Install and configure Piper Phonemize
WORKDIR $CODE_PATH/piper/lib
RUN mkdir -p Linux-arm64/piper_phonemize
WORKDIR Linux-arm64/piper_phonemize
# Use a tested version of Piper Phonemize
RUN wget https://github.com/rhasspy/piper-phonemize/releases/download/v1.1.0/libpiper_phonemize-arm64.tar.gz \
    && tar -xf libpiper_phonemize-arm64.tar.gz \
    && rm libpiper_phonemize-arm64.tar.gz \
    && cp -r $CODE_PATH/piper/lib/Linux-arm64/piper_phonemize/lib/espeak-ng-data /usr/share/espeak-ng-data

# Build and test Piper
WORKDIR $CODE_PATH/piper
RUN cmake -Bbuild -DCMAKE_INSTALL_PREFIX=install
RUN cmake --build build --config Release
RUN cmake --install build
WORKDIR build
RUN ctest --config Release

# Create a short alias for Piper
RUN echo "alias piper='$CODE_PATH/piper/build/piper'" >> ~/.bashrc \
    && source ~/.bashrc

# Create model directory and add sample voices
RUN mkdir -p $CODE_PATH/piper/voices/aru/medium

# Create a list file with each URL on a new line
RUN echo -e "https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_GB/aru/medium/en_GB-aru-medium.onnx\n \
https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_GB/aru/medium/en_GB-aru-medium.onnx.json\n \
https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_GB/aru/medium/MODEL_CARD" > aru_medium_model_file_urls.txt

# Download all URLs listed in the file
RUN wget -i aru_medium_model_file_urls.txt -P $CODE_PATH/piper/voices/aru/medium

# Return to the code path
WORKDIR $CODE_PATH

Building and Managing the Piper TTS Docker Container on Raspberry Pi 5

Building the Docker Image

The build process should take about 4–5 minutes if your system is equipped with NVMe storage. The duration can vary based on your Raspberry Pi 5’s storage performance and RAM configuration. I recommend using NVMe storage, as SD-card performance may not be sufficient and could extend the build time. However, an SD-card should still be compatible.

The Raspberry Pi 5 officially operates at 1x PCIe 2.0 speeds, delivering about 5 GT/s per lane. However, it can be configured to run at PCIe 3.0 x1 speeds, which are available on one of the four PCI Express lanes on the external connector, nearly doubling the throughput to 8 GT/s. This enhancement makes NVMe Gen 3 drives particularly well-suited for such upgraded configurations. For more detailed insights, refer to the specifications provided by specific NVMe base or drive manufacturers.

To build the Docker image piper-tts-rpi5 on your Raspberry Pi 5, run the following command in your working directory where the Dockerfile is located:

sudo docker build -t piper-tts-rpi5 .

Running the Docker Container

To run the Docker container for the first time and set its name to piper-tts, use this command:

sudo docker run --name piper-tts -it piper-tts-rpi5

Within the container

Verify the Piper Alias: Inside the container, check if the piper alias is correctly set up by running:

piper -h

Running Inference: To perform a text-to-speech inference, use the following command:

echo "To address the elephant in the room: using text-to-speech technology isn’t just practical, it’s a lot of fun too!" | piper --model /opt/piper/voices/aru/medium/en_GB-aru-medium.onnx --output_file /opt/welcome.wav

Exiting the Container:

exit

Copying the Result File to Host Machine

To copy the generated audio file from the Docker container to the host machine, execute:

docker cp piper-tts:/opt/welcome.wav ./

Managing the Container

After the initial setup, you can stop and start the container by its name as needed:

sudo docker start piper-tts
sudo docker stop piper-tts

Outside the container

Running Inference from the Host Machine: To perform text-to-speech inference without entering the container, use the following command:

sudo docker exec -ti piper-tts sh -c 'echo "To address the elephant in the room: using text-to-speech technology isn’t just practical, it’s a lot of fun too!" | /opt/piper/build/piper --model /opt/piper/voices/aru/medium/en_GB-aru-medium.onnx --output_file /opt/welcome.wav'

Copying Result Data: The result can be copied to the host as previously described with this command:

docker cp piper-tts:/opt/welcome.wav ./

Downloading the Generated .wav File

To download the generated .wav file from your Raspberry Pi, there are a few options available. One of the most convenient methods is using the SSH copy command. Execute the following command to securely transfer the file from your Raspberry Pi to your local machine, ensuring you update the related Raspberry Pi 5 hostname and paths with the actual ones:

scp rpi5-hostname:/path/to/file.wav ./local/pc/path/

This command uses SCP (Secure Copy Protocol) to transfer the file securely over SSH, making it a reliable choice for retrieving your files.

Conclusion

These basic instructions provide you with the essentials needed to get a Raspberry Pi 5 hosted Piper TTS prototype up and running. The steps are laid out in a straightforward manner, allowing you the flexibility to adjust and enhance them with additional automation, create Ubuntu users, and tweak file paths as needed. If you found this guide helpful or if it sparked some ideas for your projects, don’t hesitate to hit the like button and share it with others.

For more insights and discussions, connect with me on:

• LinkedIn
• Twitter

Thank you for reading!

Welcome to this step-by-step guide on setting up an environment to run language models such as Phi-3 or Mistral 7B on your Mac. A key advantage of this setup is that no network connection is required after the initial installation — everything runs locally on your device. This guide is tailored for CPU inference, which is ideal for most PCs lacking robust GPU capabilities. We will utilize the ONNX Runtime GenAI as the generative framework.

The latest Arm64-based CPUs — or more precisely, Systems on Chip (SoCs) — are integral to this setup, offering significant computational power to manage generative AI tasks daily. Their robust performance is particularly advantageous for efficiently running properly quantized models like the Phi-3 mini 4K instruct or the Mistral 7B instruct.

Before diving into the details, here’s a quick video showcasing what you can achieve by following this guide:

This video demonstrates the process of real-time inference for a fun task: generating a fairy tale based on the Matrix movie. It utilizes compiled ONNX Runtime libraries and a straightforward Python script, all tested on a Mac Mini M2 Pro. The instructions provided are designed to be applicable to most cases involving Macs with Arm64 architecture.

ONNX Runtime / ONNX Runtime GenAI

ONNX Runtime is a performance engine built to run machine learning models serialized in the ONNX (Open Neural Network Exchange) format. It’s compatible with models trained using frameworks such as PyTorch and TensorFlow, allowing it to function across various platforms. This engine is versatile enough to utilize CPUs, GPUs, and other hardware accelerators, making it suitable for a broad range of computing environments, from personal computers to high-end servers.

The ONNX Runtime GenAI package is a specialized version of ONNX Runtime, specifically developed for generative AI models. It is designed to handle complex tasks like image generation and natural language processing efficiently.

This guide will help you build the ONNX Runtime GenAI package from source on an Arm64-powered Mac, focusing on using the CPU for inference tasks. Although the optimizations applied during this process can sometimes lead to trade-offs in accuracy or increased inference loss, they are crucial steps toward making these models more accessible and performant on a wide range of devices. As the field continues to evolve, these optimization techniques will likely lead to even greater advancements in model efficiency without significantly compromising performance.

Python and Virtual Environment

This step is optional if you are already familiar with Python environments. Creating a virtual environment is crucial for managing software dependencies separately for different projects, allowing smooth context switching on the same machine.

Verify Python Installation

First, check if Python is installed and determine its version:

python3 -V

You should see output similar to Python 3.12.5.

Create and Activate a Virtual Environment

Next, set up a new virtual environment in your preferred directory:

python3 -m venv onnxruntime-genai-env
source onnxruntime-genai-env/bin/activate

Update pip

Ensure your pip is up-to-date to avoid any compatibility issues:

pip install --upgrade pip

By following these steps, you’ll establish a dedicated environment tailored for working with the ONNX Runtime GenAI package, effectively isolating it from other projects and dependencies.

Install Build and Dev Tools

Before proceeding, it’s important to make sure that the necessary building packages are installed on your system, particularly if this is the first time you’re setting up a development environment. These packages include compilers and libraries essential for compiling software from source. Install them using the following command:

brew install -q cmake

Next, install the development utilities that will be required throughout this article. Tools such as curl, wget, gzip, and git are indispensable for downloading sources and managing code:

brew install -q curl wget gzip git

Executing these commands equips your system with the foundational tools needed for building and managing the ONNX Runtime GenAI package. This setup ensures that all prerequisites are met for a successful installation. These tools are also beneficial for various future projects, not just for this particular build.

Get ONNX Runtime GenAI

Start by obtaining the ONNX Runtime GenAI project files from their official GitHub repository:

git clone https://github.com/microsoft/onnxruntime-genai.git onnxruntime-genai

After cloning the repository, navigate to the onnxruntime-genai directory:

cd onnxruntime-genai

To ensure you’re working with a specific, stable version, switch to version v0.3.0:

git checkout tags/v0.3.0

Verify that you’ve successfully checked out the correct version by running git status. The output should indicate HEAD detached at v0.3.0.

Note: Currently, higher versions of ONNX Runtime GenAI may require the nightly build of ONNX Runtime, adding complexity to the setup process. This situation is manageable but requires additional steps, such as installing packages via NuGet or downloading pre-compiled versions from the Visual Studio servers. If you need to work with these higher versions, prepare for a bit more setup work.

Get ONNX Runtime

You can obtain the latest version of ONNX Runtime from their GitHub releases page. Access it here: ONNX Runtime Latest Releases.

To find the download link for the version compatible with your Mac’s Arm64 architecture, use the following command:

curl -s https://api.github.com/repos/microsoft/onnxruntime/releases | grep browser_download_url | grep 'osx-arm64' | head -n 1 | cut -d '"' -f 4

This command might output a URL such as:

• https://github.com/microsoft/onnxruntime/releases/download/v1.19.0/onnxruntime-osx-arm64-1.19.0.tgz

This specific version has been tested and confirmed to work, so it’s recommended to stick with it to avoid any potential compatibility issues.

Download the specified version with:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.19.0/onnxruntime-osx-arm64-1.19.0.tgz

Extract and Setup the ONNX Runtime Files

Next, you’ll need to extract the files from the downloaded archive, onnxruntime-osx-arm64-1.19.0.tgz, into your cloned onnxruntime-genai repository. It's crucial to preserve the structure of the lib and include directories to ensure the build process runs smoothly:

Extract the archive:

tar -xzvf onnxruntime-osx-arm64-1.19.0.tgz

Move the extracted directory:

mv onnxruntime-osx-arm64-1.19.0 ort

This set of commands first extracts the files, resulting in a new directory named onnxruntime-osx-arm64-1.19.0. Moving this directory to ort renames it while keeping the internal structure (including lib and include directories) intact. Ensure these commands are executed within the directory where the onnxruntime-genai repository resides to avoid any path issues.

This setup is vital as incorrect placement of these directories can lead to build errors stating that the ONNX Runtime library could not be found.

Building

Now that you are positioned within the onnxruntime-genai directory, you can begin the build process. Execute the following command to start building:

sh build.sh --build_dir=build/macOS --config=RelWithDebInfo

This command specifies build/macOS as the build directory and sets the build configuration to RelWithDebInfo, which includes both release optimizations and debug information, balancing performance with debuggability.

Once the build process is complete, you will find the Python wheel package in the onnxruntime-genai/build/macOS/RelWithDebInfo/wheel directory. This .whl file is ready for installation.

Install the Built Package

Now that the package is built, it’s ready for installation. You will need to install both the onnxruntime_genai and the numpy packages for CPU inference. Execute the following commands:

pip3 install numpy
# Relative path within the onnxruntime_genai directory
pip3 install build/macOS/RelWithDebInfo/wheel/onnxruntime_genai-0.3.0-cp312-cp312-macosx_14_0_arm64.whl

This installs numpy, which is a dependency for many Python-based data processing and machine learning tasks, and then installs the newly built onnxruntime_genai package.

To verify the installation, you can run a simple command to check if the onnxruntime_genai package is properly imported and can invoke its models:

python3 -c 'import onnxruntime_genai; print(onnxruntime_genai.Model.device_type)'

If there are no errors and you see an output, it confirms that the onnxruntime_genai package has been successfully installed and is functioning correctly.

Download LLM Model

A ‘4K’ context length in a language model means it can handle up to 4000 tokens of text at once. This allows the model to understand and generate responses based on a larger amount of information, improving its ability to maintain coherence in longer passages or complex conversations.

Option 1: Phi-3 mini 4K instruct

To ensure efficient inference from the start, we’ll download the Phi-3 mini 4K instruct model (ONNX format). This model is particularly suited for CPU inference as it is optimized with a CPU INT4 RTN block 32 and accuracy level 4.

Phi-3-mini is a Transformer-based language model with 3.8 billion parameters, trained on high-quality, educationally valuable datasets enhanced with NLP synthetic texts and chat data from both internal and external sources to enhance chatting capabilities. Post initial training, Phi-3-mini underwent chat-specific fine-tuning using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), resulting in marked improvements in alignment, robustness, and safety.

You can find the Phi-3 mini 4K instruct model files on the Hugging Face portal. The model includes files tailored for different types of inference platforms: CPU, CUDA, and DirectML. However, we will focus solely on the CPU files, which are located here: Phi-3 CPU Files.

The model can be downloaded using huggingface-cli, git clone, or simply file-by-file with wget.

huggingface-cli (Recommended)

First, install huggingface-cli:

pip install -U "huggingface_hub[cli]"

Download only the necessary data:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4/ --local-dir ./Phi-3-mini-4k-instruct-onnx
# Download the license file
huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx --include LICENSE --local-dir ./Phi-3-mini-4k-instruct-onnx

Alternatively, you can download the entire repository:

huggingface-cli download microsoft/Phi-3-mini-4k-instruct-onnx

git clone

For handling large files within repositories, it’s essential to install and configure Git Large File Storage (Git LFS) before cloning:

Install Git LFS:

brew install git-lfs
git lfs install

Clone the repository:

git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx

wget

While downloading file-by-file with wget is possible, it is not the most efficient method. For example:

wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx/blob/main/README.md

Nevertheless, huggingface-cli remains the recommended method for its simplicity and effectiveness in downloading exactly what you need.

Option 2: Mistral 7B instruct v0.2

As an optional enhancement, consider using the Mistral 7B model as an alternative to the Phi-3. To seamlessly switch between models, download the Mistral-7B-Instruct-v0.2 ONNX model and update the model_path parameter in the inference script, which will be detailed later in this guide.

Explore the Mistral 7B model files on the Hugging Face portal. The CPU-supported version, optimized with INT4 RTN block 32 and accuracy level 4, is designed for efficient CPU inference.

The installation and download process is the same as for the previous model, although the download link differs slightly. To download the Mistral-7B model using huggingface-cli, execute the following command:

huggingface-cli download microsoft/mistral-7b-instruct-v0.2-ONNX --include onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4/ --local-dir ./mistral-7b-instruct-v0.2-ONNX
# Download the license file
huggingface-cli download microsoft/mistral-7b-instruct-v0.2-ONNX --include License.txt --local-dir ./mistral-7b-instruct-v0.2-ONNX

Note: While the Mistral 7B ONNX model is compatible with both the Phi-3 mini 4K and Phi-3 medium 128K ONNX models, as all are supported by ONNX Runtime, it may perform slower than the Phi-3 mini 4K. This reduced speed is due to its architectural differences and higher number of parameters.

Option 3: Phi-3 medium 128K instruct

The installation and download process is the same as for the previous model, though the download link differs slightly.

Phi-3-medium is a Transformer-based language model equipped with 14 billion parameters, trained on high-quality educational data and augmented with diverse NLP synthetic texts, as well as internal and external chat datasets to enhance its chat capabilities. After its initial training, Phi-3-medium underwent chat-specific fine-tuning using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), leading to substantial improvements in alignment, robustness, and safety.

huggingface-cli

Clone the repository:

huggingface-cli download microsoft/Phi-3-medium-128k-instruct-onnx-cpu

Note: The Phi-3 medium model may exhibit slower performance compared to the Phi-3 mini 4K model. This is because the Phi-3 mini contains 3.8 billion parameters, while the Phi-3 medium has 14 billion parameters. The number of parameters in a language model directly impacts its computational demands — more parameters generally result in more calculations per operation, which can slow down processing time.

SLMs (Small Language Models), such as Phi-3, are derived from larger models through techniques like model compression and knowledge distillation. Further optimization through ONNX enhances its execution efficiency, making it particularly suitable for performance-critical environments. This compact model requires significantly less computational power and data, ideal for applications where efficiency and speed are crucial. These optimizations ensure robust performance without the extensive resource demands typical of larger language models (LLMs).

Inference

Below is a streamlined chat conversation script named onnxruntime-genai-test.py, crafted to be as simple and compact as possible. It includes only the essential steps required for effective inference and does not retain any data or state from previous interactions. Ensure this script is placed in the same directory as the downloaded model, as the model path specified within the script is relative to its location. Adjust the model path according to your specific setup to ensure everything functions correctly.

# Compiled library
import onnxruntime_genai as engine

# Option 1: Phi-3 mini 4K instruct model path
model_path = "Phi-3-mini-4k-instruct-onnx/cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4"

# Option 2: Uncomment the following line to use the Mistral 7B instruct v0.2 model path
# model_path = "mistral-7b-instruct-v0.2-ONNX/onnx/cpu_and_mobile/mistral-7b-instruct-v0.2-cpu-int4-rtn-block-32-acc-level-4"

# Option 3: Uncomment the following line to use the Phi-3 medium 128K instruct model path
# model_path = "Phi-3-medium-128k-instruct-onnx-cpu/cpu-int4-rtn-block-32-acc-level-4"

print(f"Loading {model_path} model\n")

model = engine.Model(f'{model_path}')
tokenizer = engine.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()

# Set the maximum number of tokens to generate
# The full example includes additional parameters for more complex configurations
search_options = {'max_length': 2048}

# Template for formatting prompt messages in the script
chat_tpl = '<|user|>\n{input}<|end|>\n<|assistant|>'

print("Let's chat!\n")

# Chat loop. Exit with Ctrl+C
while True:
    text = input("> User: ")
    if not text:
        print("Please, answer something")
        continue

    # Populate the chat template with user input
    prompt = f'{chat_tpl.format(input=text)}'

    input_tokens = tokenizer.encode(prompt)

    gen_params = engine.GeneratorParams(model)
    gen_params.set_search_options(**search_options)
    gen_params.input_ids = input_tokens
    generator = engine.Generator(model, gen_params)

    print("\n> Assistant: ", end='', flush=True)

    try:
        # Loop to generate and display answer tokens
        while not generator.is_done():
            generator.compute_logits()
            generator.generate_next_token()
            next_token = generator.get_next_tokens()[0]
            print(tokenizer_stream.decode(next_token), end='', flush=True)
        print('\n')
    except KeyboardInterrupt:
        print("\nCtrl+C pressed, break\n")

# Delete the generator
del generator

This script is adapted from an example on the ONNX Runtime GenAI GitHub page. For more detailed information and to adjust inference parameters, refer to this example.

To run the inference script:

python3 onnxruntime-genai-test.py

To stop or exit the script, use the Ctrl+C keyboard combination.

Conclusion

That wraps up our guide on running on-device inference with the Silicon Mac. If you found this guide helpful or it inspired some ideas for your projects, don’t hesitate to hit that like button and share it around.

Connect with me for more insights and discussions:

• LinkedIn
• Twitter

Thanks for reading!

In this article, I’d like to briefly discuss these topics:

• What is technical translation under the hood?
• What can be done with the help of AI? (based on my recent project)
• Last but not least, here is a useful hint on how to effectively employ AI-powered translations.

When working with new functionality, nearly every task I tackled as a seasoned software developer, specializing in billing and payment systems for many years, involved some lexemes and translation elements. Of course, I didn’t translate the lexemes myself, but I needed to understand the logic of the process behind them.

Along with obvious things like case, declension, syntax, and grammar, there were a few more specific steps. Let me share a few things that could be potential pitfalls in the development process with translations:

• A lexeme could contain a dynamic piece of information that is not known beforehand. For example, “You have {num} apples.” But wait, what if it’s only one apple? In this case, the lexeme would probably be: “You have one apple.” The “apples” became “apple,” with numerical substitution in place, and a new lexeme is needed for the singular form.
• A lexeme might require a particular size in length. For example, a title or text for the notification we all receive tons of every day. That’s very important, especially for urgent messages. Never underestimate the importance of UX; it’s so important! Given that, the text has to be about the same length in various languages.
• In addition, translating software interfaces requires not just linguistic skills but an understanding of usability and UI/UX principles. Some or all lexemes may require a screenshot or a sketch with lexeme placement within the app. That requires more attention and time.
• Technical vocabulary. Yes, we’re all there, in a world where applications and interfaces have their own slang, if I may say so. Let us not obfuscate the essence unnecessarily; the key point here is simplicity.
• In addition to the previous statement, for accurate translation, especially in IT, knowledge of the specific domain (like networking, software development, and cybersecurity) is essential. It can be any domain, of course; this is just an example.
• Especially in technical documents, maintaining consistency in terminology and style is crucial to avoid confusion. So, a TL;DR of the translatable context may be helpful, especially with huge texts like Terms and Conditions or other complicated articles.
• As mentioned, understanding the legal and ethical implications, including copyright and data privacy issues, is important when translating content. Collaboration between, say, the legal department and translators can be very important.
• Computer-Assisted Translation (CAT) tools can help manage translations by maintaining translation memories and glossaries. Translators should be familiar with them.
• Etc.

All these things and potential issues may frighten someone who would like to create their own app, all because of the level of responsibility and time to invest in it. Luckily, with AI, such tasks could not be completely eliminated, but they become much easier, especially for free-time tinkering projects. The level of translations of the open-source app could be significantly increased.

Exempli gratia. Recently, I’ve created my own Python-powered Markdown Editor — Notolog. What have I done with translations? I created 17 translations along with the default one in English. The list of translations: Chinese (Simplified), Dutch, English, Finnish, French, Georgian, German, Greek, Hindi, Italian, Japanese, Korean, Latin, Portuguese, Russian, Spanish, Swedish, Turkish. Yes, you read it right, even Latin! Cur non, si possum? (Why not if I can?)

All these translations are available out of the box, just in time to release a beta version of the app.

Here how it looks like in the real app, Japanese translation. The main menu, toolbar elements, and status bar:

I almost ended up adding an emoji language translation, but it looked awful in the real app. Anyway, it’s possible; I’ve checked.

An interesting fact is that the output results were so good that I found myself chatting with the AI about some specific aspects related to particular languages and the intricacies of technical translation in those languages. For example, “Why is there a dash here?” and “Why is there a space before the question mark?” Such interactions build up your everyday skills as well, so do not miss such an opportunity.

To summarize the topic I’d like to describe: the incorporation of AI-powered translations could be a remedy for overwhelming tasks like complex applications and tight time slots. For sure, it has to be double-checked and moderated as much as possible because it’s more like a proof-of-concept I like to specialize in.

It will be, and it already is, quite impactful for those who work with translations, but I wouldn’t say it would be a bad thing. As a tool, it may open many more possibilities in translations themselves and the creation of new, more professional tools that will help in co-piloting without a hassle.

The Hint

Typical translations can be performed using tools like chat assistants or completion models. For my project, I began translating with ChatGPT 3.5 and continued with ChatGPT 4. There are more specific tools and AI-powered plugins for translations. However, I’d like to share a very simple example to better explain the application domain. Consider tools that can be applied to your tasks.

When translating technical text, it’s always a good idea to create a key-value pair, where the key is an unchangeable part, and the value is the text to translate. Example below:

"myapp_settings_ui_app_title": "Notolog - Python Markdown Editor",
"myapp_settings_ui_app_description": "Notolog is an open-source markdown editor developed with Python and PySide6.",
"myapp_settings_ui_font_size": "{size}pt",

The key-value pair is also convenient for retrieving value data by key, which remains constant across the different translations.

This text could be fed to your assistant chat or API. To prepare the model for the translation input, let’s set the initial prompt:

You’re a technical translation assistant with experience in user interfaces and product requirements. Take the text, which is a key-value list, and translate the values, leaving the key untouched. Values may contain a technical part surrounded by curly brackets {} that shouldn’t be changed.

Alternatively, it can be a bit more complicated and may require an example to illustrate:

You are a technical translation assistant specialized in user interfaces and product requirements. Translate the text values from English to the specified target language, leaving the keys and any technical parts inside curly brackets {} unchanged. Preserve formatting and special characters outside of the technical parts.

For instance, if the input is:

“myapp_settings_ui_app_title”: “Notolog — Python Markdown Editor”,
translate it to:

“myapp_settings_ui_app_title”: “Notolog — 파이썬 마크다운 에디터”,

Ensure the output remains in the same key-value format as the input.

After that, you may feed the text to the assistant model with the next prompt like:

Translate this text to Korean: “““ YOUR TEXT, MULTI-LINE ”””

Triple quotes create a clear start and end point for code blocks or large text segments in markdown, which is particularly useful when requesting AI assistance. Text within triple quotes is treated as preformatted text, meaning it preserves the spacing, line breaks, and indentation of the original text.

That’s all for this article. I hope it will help someone build their next app.