Deploying LLM Models on Mobile Devices with Low Power Consumption

The push to run large language models directly on phones and tablets is driven by three hard requirements: latency, privacy, and offline availability. But the physics of mobile hardware creates a ceiling. NPUs and DSPs on flagship SoCs are powerful, yet thermal design power and battery capacity turn long-context inference or multi-turn reasoning into a rapid drain. The practical path forward is not all-edge or all-cloud. It is a tiered architecture where small, quantized models handle sensitive, frequent tasks locally, and a predictable cloud API handles everything else.

Model Selection and Quantization for Mobile

To keep power draw under control, the model must fit into device DRAM without constant swapping, and the working set must be small enough to avoid sustained high-frequency memory clocks. For most current mobile hardware, this means targeting models between 1B and 4B parameters, quantized to INT4 or INT8.

Strong candidates include Llama 3.2 1B and 3B, Qwen 2.5 0.5B through 3B, Phi-3 Mini 3.8B, and Gemma 2B and 4B. These architectures use grouped-query attention or multi-query attention, which shrinks the KV cache and reduces memory bandwidth, one of the largest contributors to energy consumption on mobile SoCs.

Use quantization formats that your runtime supports natively. GGUF via llama.cpp is the most common path for rapid prototyping. For production Android apps, ONNX Runtime with INT8 QDQ graphs and Qualcomm QNN delegates lets you execute on the Hexagon NPU. On iOS, Core ML Tools converts models to use the Neural Engine.

from llama_cpp import Llama

llm = Llama(
    model_path="./qwen2.5-1.5b-q4_k_m.gguf",
    n_ctx=2048,
    n_threads=4,
    verbose=False
)
output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Summarize this paragraph."}]
)

Inference Engines and Runtime Targets

The choice of runtime determines whether you are burning watts on the CPU or executing efficiently on the NPU or GPU.

• llama.cpp. The de facto standard for mobile LLM inference. It supports Apple Metal on iOS and ARM NEON / dotprod on Android. Vulkan GPU offload is available for Adreno and Mali GPUs. It is the easiest path for GGUF models.
• MediaPipe LLM Inference API. A cross-platform Google solution that bundles model weights and handles memory mapping. It is useful if you want identical code across Android and iOS and do not need custom quantization.
• ONNX Runtime Mobile. Best when you need delegate support for hardware accelerators. The Qualcomm Neural Network (QNN) delegate runs on Hexagon NPU, and the Core ML delegate targets Apple Neural Engine. This is the path to lowest sustained power for larger sub-4B models.</