Tech Product Management

Few Lessons from Deploying and Using LLMs in Production Deploying LLMs can feel like hiring a hyperactive genius intern—they dazzle users while potentially draining your API budget. Here are some insights I’ve gathered: 1. “Cheap” is a Lie You Tell Yourself: Cloud costs per call may seem low, but the overall expense of an LLM-based system can skyrocket. Fixes: - Cache repetitive queries: Users ask the same thing at least 100x/day - Gatekeep: Use cheap classifiers (BERT) to filter “easy” requests. Let LLMs handle only the complex 10% and your current systems handle the remaining 90%. - Quantize your models: Shrink LLMs to run on cheaper hardware without massive accuracy drops - Asynchronously build your caches — Pre-generate common responses before they’re requested or gracefully fail the first time a query comes and cache for the next time. 2. Guard Against Model Hallucinations: Sometimes, models express answers with such confidence that distinguishing fact from fiction becomes challenging, even for human reviewers. Fixes: - Use RAG - Just a fancy way of saying to provide your model the knowledge it requires in the prompt itself by querying some database based on semantic matches with the query. - Guardrails: Validate outputs using regex or cross-encoders to establish a clear decision boundary between the query and the LLM’s response. 3. The best LLM is often a discriminative model: You don’t always need a full LLM. Consider knowledge distillation: use a large LLM to label your data and then train a smaller, discriminative model that performs similarly at a much lower cost. 4. It's not about the model, it is about the data on which it is trained: A smaller LLM might struggle with specialized domain data—that’s normal. Fine-tune your model on your specific data set by starting with parameter-efficient methods (like LoRA or Adapters) and using synthetic data generation to bootstrap training. 5. Prompts are the new Features: Prompts are the new features in your system. Version them, run A/B tests, and continuously refine using online experiments. Consider bandit algorithms to automatically promote the best-performing variants. What do you think? Have I missed anything? I’d love to hear your “I survived LLM prod” stories in the comments!

Sad but true. The closer the agent gets to production, the more the cracks begin to show. Teams make two mistakes at this point. They either look for a bigger/smarter LLM or endlessly iterate on prompting and RAG. Neither works. Successful agents start with the workflow, not the LLM. The more detailed the description of the workflow and outcomes, the less the agent needs to rely on AI. Every time the agent must guess what the next step is or what tools and information to use at this step, it creates an opportunity for small mistakes. They compound across multiple steps into much larger failures. Next, the workflow and the domain expertise required to deliver the outcome must be built into a knowledge graph. Trying to stuff everything into markdown files is a recipe for hallucination pie. The longer the file, the harder it is for LLMs to keep things straight. They lose focus and lose sight of what information is important. Knowledge graphs fix this by giving the agent exactly the information it needs at exactly the step it needs it. When agents get lost, and uncertainty metrics rise, the knowledge graph can deliver examples and metrics that define success and refocus the agent on iterating until it builds an acceptable output. Knowledge graphs can deliver guardrails that prevent agents from falling into endless loops. The goal is to build agents that rely on LLMs as little as possible and only deploy LLMs for what they are good at. Use the smallest models possible, and open-source models should handle over 80% of the workflow. Finally, agents need real-world feedback to improve. Version 1 is never perfect, and it takes multiple improvement cycles to be ready for deployment. Agents and knowledge graphs must be architected to benefit from improvement cycles. Every mistake creates the data required to ensure it never happens again.

Last week I presented on a panel at True Blue Blazing. The subject was Lessons Learnt from the worst projects ever.   Many people told us that they would have loved to know more so here is a quick checklist of what to do if you want your next project to succeed.   1. Don’t do it on the cheap Trying to save money by using volunteers, interns, or in-house staff who’ve never touched Salesforce before is the fastest way to triple your costs later. DIY might work for flat-pack furniture, but not for enterprise technology. If you think hiring an expert is expensive, try hiring an amateur.   2. Don’t put all your eggs in one basket The classic “Train the Trainer” trap: one super-user attends training because “everyone’s busy,” and then they leave. Suddenly, no one knows how the system works, and all the process knowledge walks out the door. Spread training across all users. Document everything. Set yourself up for success.   3. Measure seven times, cut once Discovery isn’t a luxury — it’s insurance. Spend time in the problem - don’t jump right into solutions Projects that skip proper discovery and stakeholder engagement end up rebuilding halfway through. Invest the time upfront. It’s cheaper than rework later.   4. Clear ownership When everyone thinks someone else owns it, no one really does. Projects without a captain drift — decisions stall, priorities conflict, and accountability disappears. Assign clear ownership and decision-making authority from day one.   5. Executive Buy-In If leaders aren’t visibly aligned, teams won’t be either. Mixed messages from leadership create confusion and kill adoption. When executives sing from the same hymn sheet, everyone else learns the tune.   6. Check Credentials Choosing a partner or consultant based purely on price is a false economy. Choosing the right implementation partner or consultant is one of the biggest success factors in any digital transformation project. Price does matter, but value, experience, and fit matter far more. Check credentials, reference check as you would an employee, and make sure they have current happy customers who can vouch for them.   The Moral of the Story   Every failed project leaves clues. If you don’t learn from them, you’re destined to repeat them.   Because projects don’t fail — people let them.

After two decades in technology, one pattern becomes very clear: Building a working demo is easy. Building a system that behaves reliably in production is where the real engineering begins. The same applies to LLM applications. Anyone can connect an LLM to a prompt and get an impressive response. But once that app is used by real employees, real customers, real business teams, and real workflows, the expectations change completely. Now the system needs to handle unclear inputs. It needs to retrieve the right context. It needs to avoid hallucinations. It needs to protect sensitive data. It needs to control tool access. It needs to be monitored, tested, audited, and improved continuously. That is why guardrails and monitoring are not “nice-to-have” layers. They are part of the core architecture. A serious LLM application needs more than a powerful model. It needs: A clear use case. The right model selection. A trusted RAG layer. A disciplined prompt structure. Input validation. Output validation. Tool and API controls. Observability. Evaluation. Secure deployment. In my experience, the projects that fail usually do not fail because the model was incapable. They fail because the surrounding system was not designed with enough discipline. No ownership. No risk model. No evaluation loop. No monitoring baseline. No access control. No production thinking. Enterprise AI cannot be treated like a chatbot experiment. It has to be engineered like any other critical software system. The model is only one component. The architecture around it is what determines whether the application becomes trusted, scalable, and useful in the real world. That is the shift every technology leader needs to understand.

We analyzed 70 Salesforce Commerce Cloud implementations. Only 4% achieved their promised ROI within Year 1. The difference wasn't budget, team size, or timeline. It was these 5 overlooked decisions: **1. Starting with Reference Architecture (96% success rate)** Brands that customized the Reference Architecture instead of building from scratch went live 3x faster with 60% fewer bugs. **2. API-First Integration Strategy** → Failed projects: Point-to-point integrations → Successful projects: API management layer from Day 1 Cost difference: $50K upfront, saves $500K in Year 2 **3. Phased Migration vs. Big Bang** Success rates: - Big Bang approach: 23% - Phased migration: 87% The catch? Phased takes 2 months longer but delivers value in Week 3. **4. Performance Budget Before Development** Teams that set a performance budget (2.5s page load, 500ms API response) BEFORE development achieved: - 40% better Core Web Vitals - 22% higher conversion - 50% less refactoring **5. The "Boring" Stack Wins** Successful implementations used: - OOTB features for 80% of requirements - Custom code only for competitive differentiation - Standard integrations over custom APIs - Proven patterns over latest trends The brutal reality: Every "innovative" architecture decision that failed cost an average of $400K to fix. Your stakeholders don't want innovation. They want predictable revenue growth. My advice after 10 years and 100+ implementations: Be boring with your foundation. Be innovative with your business logic. Be ruthless with your performance metrics. Save this checklist for your next SFCC project: ☐ Start with Reference Architecture ☐ Design API layer first ☐ Plan phased approach ☐ Set performance budget ☐ Default to OOTB What's your most expensive architecture mistake? 👇

Greenfield builds don't fail at launch.   They fail in the decisions nobody named before build started. I pulled ten reports published between December and June — WEF, PwC, KPMG, Bain, BCG, HBR, McKinsey, EY, Forrester, Deloitte. Different firms. Different lenses. Same throughline. → Trust, decision rights, and data readiness get decided at design.   → Not bolted on after the platform is live. McKinsey found decision rights get laid out deliberately at top-team level in M&A operating models that hold. KPMG found sponsorship absence is still the leading contributor to project failure. Deloitte found greenfield AI factories are already outperforming retrofitted environments — at the architectural level, not the feature level. The pattern across all ten: the firms recovering twice are the ones who treated architecture as a build-phase afterthought. The ones who built it right once treated it as the first decision, not the last checkbox. That's the feeling every delivery lead knows and rarely says out loud — sitting in month six, watching a "green" programme, and already knowing which decision got skipped in week one. Ten different research houses. One decision, made or missed, before the first line of the build plan gets written. 🔔 Follow Justin R. for more Enterprise Transformation insights 🚀 Subscribe my newsletter — https://lnkd.in/d_mtyWzr

Nobody tells you these things about deploying LLMs in production. I learned them the hard way, across Airtel, PwC. Here are 5 things I wish I'd known earlier: 1. Latency will surprise you more than accuracy. Your model can be brilliant and still fail in production because it takes 4 seconds to respond. At Airtel's call volumes, even 800ms matters. Optimise inference from day one not as an afterthought. 2. Prompt drift is a real problem. The prompt that works perfectly in staging quietly degrades in production as real user inputs arrive. Build prompt versioning and regression testing into your workflow like you would for any other piece of code. 3. Your vector DB choice will come back to haunt you. FAISS, Pinecone, Weaviate they all have different tradeoffs at scale. I've seen retrieval pipelines that worked beautifully at 10K documents completely fall apart at 10M. Test at production volumes early. 4. Hallucination is a product problem, not just a model problem. You can't fully eliminate it. So you design around it with guardrails, confidence thresholds, and fallback flows. The teams that win treat hallucination as a UX challenge, not just a research one. 5. Monitoring LLMs is nothing like monitoring traditional ML. There's no single metric that tells you your LLM is performing well. You need a mix latency, retrieval quality, user feedback signals, and regular human eval. Build your observability stack before you go live, not after. The gap between a working LLM demo and a production-grade LLM system is enormous. Most teams underestimate it. The ones who've shipped it don't. What would you add to this list? #LLMs #GenerativeAI #MLEngineering #AIIndia #DataScience

No offence intended when I say that I won't build anything on Salesforce until the business process is End-to-End Orchestrated and agreed on by all concerned teams. It’s tempting to start with screens, fields, and automation. But if the business process of how users will be employing the platform is not defined and agreed on, then 100 things can go wrong. 👉 Teams realise they had different expectations, features must be reworked or dropped. 👉 New requests keep popping up because the full picture wasn’t discussed earlier. And everything will be a top priority 👉 Handoff between the teams is not well defined, leading to confusion on who does what and when. 👉 Flows and code built on incomplete logic start failing 👉 Rework takes time. Missed scenarios need extra dev cycles, which weren’t planned for. 👉 The biggest fallout "Low User Adoption" - When the system doesn’t match how people actually work, they avoid using it, or create workarounds. But if the business owners are adamant about getting started without the process mapping, then you have to push back 👉 Don’t Rush to Build - inform the drawbacks of starting without proper process mapping. 👉 Request for a Workshop with all teams - Bring all key stakeholders into the same room. Walk through the process, start to finish. 👉 Document the End-to-End Flow - Create a simple process map. Highlight handoffs, pain points, and decision points. 👉 Ask for confirmation of agreed flow via a process map - Once aligned, get buy-in via email or shared doc. This avoids future backtracking. 👉 Translate Process to designs and to Salesforce feature mapping - Take it to the drawing board and map Salesforce features to each step. 👉 Build iteratively, take feedback more frequently and keep improving. Yes, Salesforce is simple to configure, but it should not be assumed that the rules of a good project execution don't apply to Salesforce Project. Thoughts? #Salesforce #SalesforceProject #SalesforcePartner #SalesforceImplementation #Anvisol

Before you sign another Salesforce SOW, ask these 5 questions. Most of the “rescue” projects I do for businesses have the same backstory: “We already spent a chunk on Salesforce… and it still isn’t how we run the business.” The damage usually starts before anyone logs in. Before you approve the next round of work, ask: 1. What exact problem are we solving in the next 4–6 weeks? - “Get more out of Salesforce” is not a problem. - “Increase Salesforce adoption from 30% to 80%,” or “Get forecast accuracy within 5–10%” is. 2. What will we stop doing outside Salesforce once this is done? - Which spreadsheets disappear? - Which Word / Excel quotes and contracts go away? - If nothing is being retired, nothing is really changing. 3. Who owns the outcome internally? - Not “IT”, not “everyone”. - One person who cares about adoption, accuracy and behaviour change, not just tickets and features. 4. How will reps and ops be forced to use Salesforce day‑to‑day? - What views will they live in? - What becomes easier in Salesforce than in email/Excel? - What breaks if they don’t use it? 5. What does success look like in one screenshot? - One dashboard leadership actually uses. - If your partner can’t describe that dashboard before starting, you’re buying hours, not outcomes. On a recent client, we ran through exactly these questions and then: - Rebuilt the process around how the business actually sells - Killed the shadow spreadsheets and made Salesforce the single source of truth - Tightened stages and data so the forecast stopped being an “educated guess” Six weeks later: adoption went from ~20% to 85%, and forecasting landed within 5–10% of reality. Same licences. Same team. Different questions at the start. If you’ve got a Salesforce project on the table and you’re not confident in the answers to those 5, comment “BLUEPRINT” or DM me “Blueprint.” I’ll send you a link to book a 15‑minute Salesforce Blueprint call where we’ll: - Pressure‑test your current plan against these questions - Spot the gaps that usually lead to messy, half‑used orgs - Outline what a focused 4‑week fix should actually target for your team No slides. Just me, your Salesforce, and a blunt sanity check before you spend another pound on it.

“We implemented Salesforce, but it doesn’t work for our organization.” My response is always the same: Salesforce isn’t failing. The implementation is. A CRM is only as good as the strategy behind it. When it doesn’t deliver results, the root cause is rarely the platform itself. It’s usually one (or more) of these: • The business processes were never fully understood before implementation. • Requirements were documented based on assumptions instead of real business challenges. • The implementation partner lacked the experience to design a scalable solution. • The project was awarded to the lowest bidder, leaving little time for discovery, architecture, and user adoption. • Teams were never properly trained, so adoption remained low. • The solution was built for today’s problems instead of tomorrow’s growth. Here’s something worth thinking about. Companies like Google, Amazon, Toyota, Coca-Cola, and thousands of other global enterprises use Salesforce. Do you really think they couldn’t build their own CRM? Of course they could. They choose Salesforce because building software is only one part of the equation. A mature platform, continuous innovation, security, integrations, AI capabilities, and a global ecosystem are far more valuable than maintaining a custom CRM. The question is never: “Is Salesforce capable?” The real question is: “Are we utilizing Salesforce the right way?” One mistake I see repeatedly is this: Organizations spend crores on Salesforce licenses, but hesitate to invest a few lakhs in proper architecture and implementation. That’s like buying premium land and then trying to save money on the foundation. A weak foundation creates expensive problems for years. A strong foundation creates a system that scales with the business, improves productivity, increases adoption, and delivers measurable ROI. Technology doesn’t transform businesses. Well-designed solutions do. As a Salesforce Architect, I’ve learned that the most successful CRM projects begin long before the first line of configuration or code. They start with understanding the business, challenging assumptions, designing for scale, and focusing on user adoption not just delivering features. Your CRM isn’t an expense. It’s a long-term business asset. Build it like one. #CloudIntellectSystems #Salesforce #CRM #DigitalTransformation #SalesforceArchitecture #BusinessTransformation #CustomerSuccess #SalesforceConsulting #EnterpriseArchitecture #CloudComputing #TechLeadership