Generative AI in Scale, Ratelimits, and… Peruvian cuisine
Picture this: a new restaurant has just opened in your city, serving a brand new dish. You went there by yourself, tried, and you really liked it. Then you decide to bring your family and friends to experience it. But when you arrive, the restaurant is crowded, and there are no tables available for your group.
You have several options: you can keep asking if a table is available, wait in line, look for the same dish at another location, try a different restaurant (with possibly different flavors), or even give up and apologize to your guests. In any case, your experience — and possibly your reputation — is at stake. You realize your individual visit is entirely different from scaling it up to accommodate many people.
From the restaurant’s perspective, the challenge is also significant: What is the ideal number of tables? Is it worth expanding? Should they offer reservations or create a VIP line? Should they collect customers’ phone numbers to notify them when a table is available or hand out devices to let them know when they are available? All these decisions impact costs, competition, and, of course, customer satisfaction.
But I’m not just talking about restaurants.
When scaling the use of Generative AI, a common discussion revolves around managing API Rate Limits — the number of calls that can be made to an API within a given time frame and the number of tokens processed per minute. Much like a busy restaurant, there are several potential solutions to handle this scenario. And just like a restaurant connoisseur seeking the best dining experience, finding the best solution requires time, analysis, and it won’t always be simple or immediate. This optimization requires architecture decisions and trade-offs will be made.
The first error message we often see when your Generative AI prototype starts to succeed and scale is this:
And this is by design. It’s a restriction technique that has existed in APIs long before the generative AI boom. These limits are used to prevent server overload, maintain service stability, and ensure proper API usage. While the implementation may vary, all providers set limits:
Azure OpenAI Quotas: Azure OpenAI sets limits such as requests per minute, tokens per request, and concurrent jobs, varying by model and subscription. Quotas can be increased based on resource usage.
OpenAI Rate Limits: OpenAI sets rate limits based on subscription level and calls per minute, designed to maintain consistent performance and prevent server overload.
Yes, the OpenAI service available through Azure follows different rules compared to using OpenAI directly.
Gemini API Pricing: Google’s Gemini API offers free and pay-as-you-go models. The paid model allows up to 1,000 requests per minute and processes millions of tokens, with pricing depending on the number of tokens processed and the model type.
AWS Bedrock Quotas: AWS sets quotas for its Bedrock service, with daily and monthly limits on requests and tokens. Adjustments are possible based on customer usage, enabling efficient scalability of AI services.
Well, how does this connect to a restaurant? In my view, the techniques for handling these restrictions are quite similar to the scenario where you take your guests to a restaurant, and there’s not enough space for everyone.
In the table below, I draw this analogy, highlighting how this comparison helps us understand that choosing the ideal solution isn’t simple — there are many options involved. Moreover, a large-scale GenAI solution must carefully consider these factors (and a combination of them) to best meet demand.
Scaling GenAI solutions mirrors the complexities of going to restaurants and each decision involves a unique set of trade-offs beginning with which restaurant/LLM to choose and what options they offer. Just like finding creative ways to seat your guests, architects must balance rate limits, cost efficiency, and user experience. The options are many (and are growing as each day goes on), and the ideal solution depends on the specific scenario, with each choice offering different advantages, disadvantages, and experience. Deciding which solution to implement requires not only the cost of implementation and but also the cost of LLM APIs in a high volume/high frequency scenario.
Get Leandro’s stories in your inbox
Join Medium for free to get updates from this writer.
But what about the Peruvian Cuisine?
Recently, I traveled to Lima, Peru, and was able to visit many different places. Lima is known as one of the world’s top gastronomic destinations, thanks to its rich, diverse, and innovative cuisine. Peruvian cuisine is famous for combining indigenous, Spanish, African, Japanese, Chinese, and Italian influences, resulting in a fusion of flavors and techniques. Nikkei cuisine, for example, blends Japanese and Peruvian traditions. I experienced some of the situations I’ve listed, which served as inspiration for this post.
- A callback device after you order in a Sangucheria.
2. The huge queues in front of the tiny, alway busy and famous Al Toke Pez restaurant, known by the Netflix documentary on Street Food.
3. Choosing from multi-options in San Martin’s market, trying the Chifa (a Peruvian-Chinese culinary tradition that combines Cantonese cooking techniques with Peruvian ingredients and cooking methods)
4. A reservation for a Lomo Saltado working really well
5. Cooking my own Ceviche and Causa Limeña in Luchito’s
6. A continous stream of food into my plate :)
7. Getting a spot to the famous Nikkei restaurante Maido for September (I was there in July). Too bad I had to come back earlier! But I will try again for sure. They also offer a place in the bar, but it was already full.
So that is a summary of my experiences, either on Peruvian food adventures or with the techy side of GenAI scaling. Hope you enjoyed it! If you ever want some tips on where to eat in Peru or need help with those tricky rate limits and how to scale GenAI, just let me know — I would be happy to share more!
