Interaction Cost Minimization

The Idle Improvement : Agentic Graph RAG Systems with Sleep-Time Compute 💥 Sleep-time compute represents a natural evolution in the pursuit of computational efficiency for RAG systems. Traditional approaches to improving RAG performance have focused on either refining model architectures or scaling computation at test-time. However, these approaches often result in increased latency and costs. Sleep-time compute introduces a new dimension of optimization by recognizing that when computation happens is just as important as how much computation occurs. The heterogeneous graph structures employed by both NodeRAG and PIKE-RAG create ideal foundations for sleep-time optimization. During idle periods, these systems can transform basic knowledge graphs into semantically rich networks through operations like centrality analysis, community detection, relationship inference, and graph consolidation. NodeRAG's approach of typing nodes as semantic units (S), entities (N), relationships (R), and attributes (A) provides a structured framework that can be progressively enhanced. The correlation between query predictability and sleep-time compute effectiveness has profound implications for system design. When queries are highly predictable from the context, sleep-time compute offers substantial benefits. Conversely, for less predictable queries that introduce new elements or require multiple reasoning steps beyond what was explicitly stated, the advantages diminish. By performing expensive computations once during idle periods and then reusing the results across multiple queries, systems can drastically reduce the average cost per interaction. This creates a virtuous cycle where the system becomes more cost-effective as usage increases. Test-time tokens are considered 10× more expensive than sleep-time tokens (reflecting the higher cost of latency-sensitive computation). Under this model, with 10 queries per context, sleep-time compute reduces the average cost per query by approximately 2.5× while maintaining or improving accuracy. Technical implementation of sleep-time compute shows a sophisticated approach using specialized agents and memory systems. The "sleep time memory agents" are designed to think deeply about partial information, storing inferences in memory blocks that can later be accessed by agents. This approach mirrors cognitive processes in human learning, where initial exposure to information triggers background processing that continues even when conscious attention shifts elsewhere. The process includes triggering inference through specialized prompts, deep thinking about the partial information, memory management through dedicated functions, and completion signals.

“This sounds incredible voice AI is cool & all but there’s no way we can afford this at scale. We’d be burning thousands of dollars every month—just on onboarding. There has to be a better way.” That’s what one of our B2B SaaS clients told us when they realized how expensive their AI-driven voice onboarding would be. They wanted a smooth, voice-to-voice AI assistant that could onboard users in a 5-7 minute conversation, but using OpenAI’s real-time voice streaming API was just too expensive. At $1.2-$1.5 per minute, they were looking at $5-7 per user session—scaling that up meant tens of thousands of dollars in API costs every month. So, we had a choice: tell them it wasn’t possible within their budget or build a smarter solution from scratch. We chose the latter. Here’s “How We Rebuilt OpenAI’s Voice Streaming—At a Fraction of the Cost” Instead of relying on expensive APIs, we deconstructed the voice interaction process and optimized it: 🔹 Speech-to-Text? We skipped the paid APIs and used [REDACTED] (a hidden gem for browser STT) and [REDACTED] for mobile—both run locally, for free. 🔹 AI Responses? No need for a voice-to-voice API. We fed the text into an LLM (OpenAI, LLaMA, or similar), which is way cheaper than streaming real-time voice. 🔹 Text-to-Speech? Instead of full realtime voice streaming, we used [REDACTED] to generate responses only when needed, significantly cutting costs. The Result? ✅ 80% Cost Reduction – The client went from prohibitively expensive API costs to a sustainable, scalable voice experience. ✅ Faster Response Time – With STT running locally, we reduced latency and server load. ✅ More Control – No vendor lock-in. We could self-host TTS later, further driving costs down. By the end of it, the client had the same seamless AI onboarding experience—but without the insane price tag. If you’re building AI-driven voice interactions, don’t assume you need expensive third-party APIs. Sometimes, breaking things down and rethinking the approach can save thousands while improving performance. If you're a retail, ecom company SaaS struggling with AI challenges, let's talk. Drop me a message! 📩

We spent a month on building a fantastic LLM app that was totally unaffordable and therefore useless. We learned a few things: Boring but important. How much is your LLM app costing you per user per month? Can you afford that? Many LLM apps such as chatbots require lots of LLM calls - these increase as the conversation progresses. The costs can quickly add up. If you are building for a consumer app, this is a problem. Typically you have a large number of users and are charging a small amount per user, which can make the cost of LLM calls too much to sustain. Cherrypick experimented with chat-based grocery tools in 2023. We had a proof of concept that allowed customers to add groceries to their basket via WhatsApp. It worked well and was fun and quick to use. However we realised that to provide a good service we would need to make hundreds of calls per shop, which would have entirely eroded our profit margin from that shop. Deploying it would lose us money. A meal generator as it is significantly more cost-effective for a customer that a grocery chatbot. It requires only a few LLM calls per meal plan generation, versus dozens of messages in a typical chat session. The limited number of meal plans a customer will generate each week makes it financially viable. Whatever you are building, budget for LLM calls as part of your initial investigation. Understand the number of tokens you will need to generate for each user interaction and how many interactions will be needed per billing cycle. You're probably building with OpenAI - can you use a cheaper LLM for get 80% of the quality for a tenth of the cost?

If you have a working product/service that relies on language models in the backend, there are a few very easy ways to reduce costs without sacrificing quality: (a) Creating a small custom router to decide which language model to use (you can rely on cheaper models for simpler responses); (b) Shortening a large chunk of the conversational history as your chat progresses;  (c) Caching your outputs. There are, of course, many more. Having worked with a few companies whose business model is based on having high LLM output quality while minimizing associated costs, I’m surprised that these strategies are not yet standard.

Man**l data entry feels harmless. Until you do the math. "It only takes a few minutes to copy this information over." Sure. But a few minutes times how many entries times how many systems? Here's an example: → 200 customer interactions per month → 3 minutes average to manu**ly update systems → 10 hours monthly spent on pure data duplication → $400 monthly cost (at $40/hour) → $4,800 annual cost For ONE person doing ONE type of data entry. Now multiply that across your entire team and all the different types of information that gets entered multiple places. The numbers get ugly fast. But the opportunity cost is even higher: Those 10 hours monthly could be spent on: → Actual customer service improvements → Process optimization projects → Strategic business development → Skills training and development Man**l data entry doesn't just cost money. It costs progress. Every minute spent copying information is a minute not spent growing the business. Integration isn't a luxury. It's basic business efficiency.