Audience
AI infrastructure engineers and developers seeking a fast, private local model inference and an OpenAI-compatible backend for applications and coding agents
About BaseRT
BaseRT is a high-performance LLM inference runtime for Apple Silicon that lets developers pull models from Hugging Face, chat with them locally, or serve an OpenAI-compatible API from one CLI. Accelerated by hand-written Metal kernels, it is designed to deliver fast prefill and decode performance on M-series Macs, with published benchmarks showing up to 6.4× faster prefill than llama.cpp, 3.9× faster than MLX, and up to 1.33× faster decode. The basert CLI handles model downloading, conversion, interactive chat, serving, completion, benchmarking, inspection, and bundle signing. Its server supports chat, completions, embeddings, transcription, tool calls, continuous batching, paged KV caching, and prefix caching, while supported models can process text, vision, and audio. BaseRT uses its own .base model format with Q2–Q8 affine quantization, optional AWQ calibration, and signed bundles, and can convert GGUF, Hugging Face, and MLX checkpoints.
Other Popular Alternatives & Related Software
LMCache
LMCache is an open source Knowledge Delivery Network (KDN) designed as a caching layer for large language model serving that accelerates inference by reusing KV (key-value) caches across repeated or overlapping computations. It enables fast prompt caching, allowing LLMs to “prefill” recurring text only once and then reuse those stored KV caches, even in non-prefix positions, across multiple serving instances. This approach reduces time to first token, saves GPU cycles, and increases throughput in scenarios such as multi-round question answering or retrieval augmented generation. LMCache supports KV cache offloading (moving cache from GPU to CPU or disk), cache sharing across instances, and disaggregated prefill, which separates the prefill and decoding phases for resource efficiency. It is compatible with inference engines like vLLM and TGI and supports compressed storage, blending techniques to merge caches, and multiple backend storage options.
Learn more about LMCache
Muse Spark Open
Muse Spark Open is an open-weight, natively multimodal language model for developers and organizations building AI applications, coding systems, and agentic workflows. The released model has 816 billion total parameters, 42 billion active parameters, and a 1-million-token context window. Teams can download the model weights, run inference through supported engines such as vLLM, SGLang, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp, or Unsloth, and customize the model using PyTorch. It supports long-context reasoning, software engineering, coding, tool use, and multimodal inputs. Deployment options include cloud inference through listed API providers or self-hosting on compatible GPU systems; Meta documents a desktop setup using two Apple M5 Ultra Mac Studio computers. The model is released under the Muse Model License with limited commercial terms.
Learn more about Muse Spark Open
Lucebox
Lucebox is a plug-and-play computer built for running local AI models and agents at full speed. Inside the custom chassis, a Ryzen AI MAX+ 395 with 128GB of unified LPDDR5X memory is paired with an RTX 3090, and the two work together through an open-source inference engine hand-tuned for exactly this hardware.
The architecture is what makes it fast. Large models live in the 128GB unified memory tier, while the 3090's high-bandwidth VRAM acts as a fast tier. Speculative decoding (DFlash) and speculative prefill (PFlash) bridge the two, producing inference speeds up to 10x higher than llama.cpp on the same silicon and beating machines like the Mac Studio and DGX Spark at a fraction of their effective cost.
Learn more about Lucebox
DeepInfer
DeepInfer is an open-source inference framework for running DeepSeek language models on Linux systems with supported NVIDIA or Huawei Ascend accelerators. It provides offline inference and runtime components for model execution, optimized kernels, distributed parallelism, asynchronous batch scheduling, speculative decoding, deterministic generation, and structured outputs such as JSON Schema, regular expressions, and EBNF formats. The project documents DeepSeek-V4.1-Flash support and includes build instructions, configuration guidance, examples, and evaluation tools. It is intended for developers and infrastructure teams building or evaluating self-managed inference deployments. Its repository notes that production services may require additional KV-cache, prefill/decode disaggregation, and load-balancing components not included in this release.
Learn more about DeepInfer
Company Information
Base Compute
Founded: 2026
Australia
www.basecompute.co/getbasert
Videos and Screen Captures
Other Useful Business Software
Train ML Models With SQL You Already Know
Build and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
Product Details
Platforms Supported
Mac
Training
Documentation
Support
Online
BaseRT Frequently Asked Questions
Q: What kinds of users and organization types does BaseRT work with?
BaseRT works with these users and organization types: Mid Size Business, Small Business, Enterprise, Freelance, Nonprofit, Government, and Startup.
Q: What languages does BaseRT support in their product?
BaseRT supports these languages: English.
Q: What kind of support options does BaseRT offer?
BaseRT offers support via online.
Q: What other applications or services does BaseRT integrate with?
BaseRT integrates with: Gemma 3, Gemma 4, Hugging Face, Llama 3.1, Llama 3.2, Mistral AI, OpenAI, Phi-3, Qwen3.5, Qwen3.6, and Qwen3.
Q: Does BaseRT have an API?
Yes, BaseRT has an API.
Q: What type of training does BaseRT provide?
BaseRT provides training in the form of documentation.
