Member-only story
Building a Local LLM Server: How to Run Multiple Models Efficiently
I’m writing a series on how to utilize a home Ubuntu server as a local LLM server. While working on LLM projects, I’ve pondered how individual developers or small teams can easily use LLM inference at a low cost. In most cases, the initial phase involves using services like OpenAI or Anthropic to test whether ideas function effectively, followed by processing large amounts of data using a local LLM server. Once performance expectations are established, developers often return to OpenAI or Anthropic for improved performance. Therefore, setting up a local LLM server effectively is beneficial to save costs.
In this post, I will discuss configuring a server to use multiple models, as illustrated in the above diagram. If you’re using Ollama, you don’t need to worry about these concerns, as it handles everything for you. However, using a service that hides various complexities like Ollama might hinder learning, so I’ve chosen to use llama.cpp.
To understand the issues, let’s consider the following example. In most cases, when using a local LLM with Python, you create a client using the OpenAI API, setting prompts and configuration parameters according to OpenAI’s method to request inference. Below, I will specify the model as “llama3.1” and send messages in a chat format. (Obviously, the original OpenAI API…
