Tech Product Management

Few Lessons from Deploying and Using LLMs in Production Deploying LLMs can feel like hiring a hyperactive genius intern—they dazzle users while potentially draining your API budget. Here are some insights I’ve gathered: 1. “Cheap” is a Lie You Tell Yourself: Cloud costs per call may seem low, but the overall expense of an LLM-based system can skyrocket. Fixes: - Cache repetitive queries: Users ask the same thing at least 100x/day - Gatekeep: Use cheap classifiers (BERT) to filter “easy” requests. Let LLMs handle only the complex 10% and your current systems handle the remaining 90%. - Quantize your models: Shrink LLMs to run on cheaper hardware without massive accuracy drops - Asynchronously build your caches — Pre-generate common responses before they’re requested or gracefully fail the first time a query comes and cache for the next time. 2. Guard Against Model Hallucinations: Sometimes, models express answers with such confidence that distinguishing fact from fiction becomes challenging, even for human reviewers. Fixes: - Use RAG - Just a fancy way of saying to provide your model the knowledge it requires in the prompt itself by querying some database based on semantic matches with the query. - Guardrails: Validate outputs using regex or cross-encoders to establish a clear decision boundary between the query and the LLM’s response. 3. The best LLM is often a discriminative model: You don’t always need a full LLM. Consider knowledge distillation: use a large LLM to label your data and then train a smaller, discriminative model that performs similarly at a much lower cost. 4. It's not about the model, it is about the data on which it is trained: A smaller LLM might struggle with specialized domain data—that’s normal. Fine-tune your model on your specific data set by starting with parameter-efficient methods (like LoRA or Adapters) and using synthetic data generation to bootstrap training. 5. Prompts are the new Features: Prompts are the new features in your system. Version them, run A/B tests, and continuously refine using online experiments. Consider bandit algorithms to automatically promote the best-performing variants. What do you think? Have I missed anything? I’d love to hear your “I survived LLM prod” stories in the comments!

🏗 How To Tackle Large, Complex Projects. With practical techniques to meet the desired outcome, without being disrupted or derailed along the way ↓ 🤔 99% of large projects don’t finish on budget and on time. 🤔 Projects rarely fail because of poor skills or execution. ✅ They fail because of optimism and insufficient planning. ✅ Also because of poor risk assessment, discovery, politics. 🎯 Best strategy: Think Slow (detailed planning) + Act Fast. ✅ Allocate 20–45% of total project effort for planning. ✅ Riskier and larger projects always require more planning. ✅ Think Right → Left: start from end goal, work backwards. ✅ For each goal, consider immediate previous steps/events. ✅ Set up milestones, prioritize key components for each. ✅ Consider stakeholders, users, risks, constraints, metrics. 🚫 Don’t underestimate unknown domain, blockers, deps. ✅ Compare vs. similar projects (reference class forecasting). ✅ Set up an “execution mode” to defer/minimize disruptions. 🚫 Nothing hurts productivity more than unplanned work. Over the last few years, I've been using the technique called “Event Storming” suggested by Matteo Cavucci to capture user’s experience moments through the lens of business needs. With it, we focus on the desired business outcome, and then use research insights to project events that users will be going through towards that outcome. On that journey, we identify key milestones and break user’s events into 2 main buckets: user’s success moments (which we want to dial up) and user’s pain points or frustrations (which we want to dial down). We then break out into groups of 3–4 people to separately prioritize these events and estimate their impact and effort on Effort vs. Value curves (https://lnkd.in/evrKJUEy). The next step is identifying key stakeholders to engage with, risks to consider (e.g. legacy systems, 3rd-party dependency etc.), resources and tooling. We reserve special timing to identify key blockers and constraints that endanger successful outcome or slow us down. If possible, we also set up UX metrics to track how successful we actually are in improving the current state of UX. When speaking to business, usually I speak about better discovery and scoping as the best way to mitigate risk. We can of course throw ideas into the market and run endless experiments. But not for critical projects that get a lot of visibility — e.g. replacing legacy systems or launching a new product. They require thorough planning to prevent big disasters and urgent rollbacks. If you’d like to learn more, I can only highly recommend "How Big Things Get Done" (https://lnkd.in/erhcBuxE), a wonderful book by Prof. Bent Flyvbjerg and Dan Gardner who have conducted a vast amount of research on when big projects fail and succeed. A wonderful book worth reading! Happy planning, everyone! 🎉🥳

After working with 5 tech unicorns, I’ve noticed something. The happiest companies? The ones with clean Salesforce orgs, consistent reporting, and reps who actually follow process... All do one simple thing: They think before they build. Every. Single. Time. Unhappy companies? They react to a problem and build immediately. • Random fields. • Random automations. • Random dashboards. And randomness eventually becomes revenue loss. Here’s the habit that fixes everything: 1. Define the problem on paper 2. Map the steps of the process 3. Decide who owns each step 4. Decide when automation matters 5. Only then open Salesforce It’s not a fancy tool. It’s not a huge project. It’s a discipline. And in some cases takes you like 30 mins to do. But that 30 minutes saves you: • 6 wasted months • 10 broken automations • Hundreds of manual tasks • Thousands in consultant fees • And millions in lost pipeline Salesforce is powerful BUT only if you give it a brain. The brain comes first. The build comes second.

Sad but true. The closer the agent gets to production, the more the cracks begin to show. Teams make two mistakes at this point. They either look for a bigger/smarter LLM or endlessly iterate on prompting and RAG. Neither works. Successful agents start with the workflow, not the LLM. The more detailed the description of the workflow and outcomes, the less the agent needs to rely on AI. Every time the agent must guess what the next step is or what tools and information to use at this step, it creates an opportunity for small mistakes. They compound across multiple steps into much larger failures. Next, the workflow and the domain expertise required to deliver the outcome must be built into a knowledge graph. Trying to stuff everything into markdown files is a recipe for hallucination pie. The longer the file, the harder it is for LLMs to keep things straight. They lose focus and lose sight of what information is important. Knowledge graphs fix this by giving the agent exactly the information it needs at exactly the step it needs it. When agents get lost, and uncertainty metrics rise, the knowledge graph can deliver examples and metrics that define success and refocus the agent on iterating until it builds an acceptable output. Knowledge graphs can deliver guardrails that prevent agents from falling into endless loops. The goal is to build agents that rely on LLMs as little as possible and only deploy LLMs for what they are good at. Use the smallest models possible, and open-source models should handle over 80% of the workflow. Finally, agents need real-world feedback to improve. Version 1 is never perfect, and it takes multiple improvement cycles to be ready for deployment. Agents and knowledge graphs must be architected to benefit from improvement cycles. Every mistake creates the data required to ensure it never happens again.

I’ve been working as a contractual Program/Project Manager on complex projects for the past 7 years, most of which followed Agile methodologies. While the Software Development Life Cycle (SDLC) is designed to reduce risk, poor implementation can have the opposite effect. If not executed properly, it significantly increases the risk of project failure. Here’s a quick ranking of critical failure points that commonly derail software projects: 🔴 1. Unclear or Changing Requirements Poorly defined needs or constant scope changes break alignment early and often. ✅ Fix: Involve stakeholders early, use user stories and clarify DoD (definition of done), and validate frequently; another advice: make sure to define change request in the initial contract with the client. 🔴 2. Inadequate Planning & Estimation Unrealistic timelines or budgets create pressure that leads to shortcuts and burnout. ✅ Fix: Buffer for unknowns, involve tech leads in estimation. 🟠 3. Ineffective Communication Team silos and misalignment cause costly rework and delays. ✅ Fix: Daily stand-ups, shared documentation, clear ownership. The tech team needs to understand the functional requirement to be able to implement it technically. 🟠 4. Weak Design & Architecture Hasty or shortsighted technical decisions lead to rework and scalability issues. ✅ Fix: Involving a software architect who could support drafting the best scalable architecture choices within the available projects needs, constraints and budget 🟠 5. Insufficient Testing & QA Testing cut short = bugs in production, bad UX, security holes. ✅ Fix: Invest in a QA strategy to identify tests to be run by type of release, and automate critical time-consuming tests 🟡 6. Lack of Stakeholder Involvement Software built in isolation rarely meets business goals. ✅ Fix: Demo regularly (ideally after each milestone), build feedback into the cycle. 🟡 7. Poor Change & Config Management Inconsistent environments and chaotic updates derail progress. ✅ Fix: Version control, CI/CD, and clear change protocols. 🟡 8. Inadequate Risk Management Unexpected issues become blockers when risks aren't flagged early. ✅ Fix: Ongoing risk logs, contingency planning. 🟢 9. Neglecting Post-Launch Support No plan for support = user churn and poor adoption. ✅ Fix: Monitor performance, address issues fast. 🟢 10. Lack of DevOps & Automation Manual processes delay releases and increase error rates. ✅ Fix: Embrace CI/CD and infrastructure-as-code. Strong software isn’t just about great code—it’s about clarity, communication, and continuous feedback. A strong Project Manager implements the right processes and follows each step methodically to spot weak links early and address them proactively. And when issues do arise (as they often do), they stay calm, communicate transparently, and ensure all stakeholders remain aligned throughout the journey. #SoftwareDevelopment #SDLC #TechLeadership #ProjectManagement #Agile #DevOps #ProductDelivery

It was 9:00 AM on a Tuesday in an Abu Dhabi high-rise. I was staring at a methodology binder thick enough to prop open a fire door. The PMO had dedicated tooling and an entirely certified team. But 12 of their last 15 projects had missed their delivery dates. I spent the first week reading their status reports. They were immaculate... ...and completely useless. Every project stayed green right up until the day it crashed. Earlier in my career, I thought a heavy methodology guaranteed execution. I believed a clean status report meant the project was healthy. I chose the safety of the template over the haqeeqat (ground reality) of the work. This month, I was diagnosing another stalled enterprise portfolio. And I realized that massive organizations do the exact same thing. We reward teams for looking good instead of being useful. Nobody gets credit for raising an early flag. They get credit for keeping their lane clean. When your culture teaches people to protect the report instead of the project, your PMO fails. It becomes a highly paid administrative burden. You must fix the incentives before you redesign the process. Here is the playbook we use to rebuild PMO culture: 𝐑𝐞𝐰𝐚𝐫𝐝 𝐭𝐡𝐞 𝐄𝐚𝐫𝐥𝐲 𝐑𝐞𝐝: Stop punishing PMs who report delays. Actively praise the first person to flag a blocker during the steering committee. Punishing early warnings guarantees catastrophic late-stage failures. 𝐊𝐢𝐥𝐥 𝐭𝐡𝐞 𝐆𝐫𝐞𝐥𝐥𝐨𝐰 𝐒𝐭𝐚𝐭𝐮𝐬: A project is either on track or it is at risk. Ban hybrid reporting colors that soften the truth. Allowing ambiguous statuses lets failing projects hide until they are unrecoverable. 𝐀𝐮𝐝𝐢𝐭 𝐟𝐨𝐫 𝐀𝐜𝐭𝐢𝐨𝐧: Do not review a status report for its formatting. Review it for the management decisions it forces. A beautiful report that triggers no executive action is just expensive typing. You cannot fix a culture of fear with a new software tool. Whether it is a global enterprise portfolio, or a heavy binder in Abu Dhabi. Khallas.

Stop Projects Disasters Before they Start A lot of projects don’t fail halfway. They start failing even before they even begin. I’ve seen it. You’ve probably seen it too. A “sure shot” project falls apart in slow motion. I was talking to my coachee Dan, a senior executive of a mid-sized Company. Six months earlier, he led a software migration for his finance team. He had executive buy-in. A solid vendor. A realistic timeline. Three weeks before launch, his CFO casually mentioned she needed two extra weeks to train her team. Nobody had planned for that. The answer isn’t more meetings. Or more planning. Smart teams anticipate problems. Here’s what actually works. Run a premortem. Get your team in a room. Ask one question: “It’s six months from now. The project failed spectacularly. What could have gone wrong?” Set a timer for 15 minutes. Write down all the ideas generated. You’ll hear things like: “The vendor’s software can’t handle our data.” “Legal never approved the contract in time.” “Sarah left and took all the core knowledge with her.” These aren’t random worries. They’re signals your team has already noticed but never voiced. Smart teams don’t just plan for success. They prepare for failure too. So try this tomorrow. Pick your biggest upcoming project. Set a 15-minute timer. - Ask your team to list every way it could fail. You could also run an AI model trained on projects executed earlier, to predict likely challenges. You’ll spot the real risks hiding behind your optimism. And you’ll fix them or have a risk mitigation plan before they derail everything. The best project wins? - They’re the disasters no one ever anticipated — because you stopped them early.

𝗜𝘀 𝘆𝗼𝘂𝗿 𝗟𝗟𝗠 𝘁𝗿𝘆𝗶𝗻𝗴 𝗵𝗮𝗿𝗱 𝘁𝗼 𝗮𝗴𝗿𝗲𝗲 𝘄𝗶𝘁𝗵 𝘆𝗼𝘂? 𝗜𝗳 𝘆𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹 𝗻𝗲𝘃𝗲𝗿 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲𝘀 𝘄𝗶𝘁𝗵 𝘆𝗼𝘂, 𝗶𝘁’𝘀 𝗻𝗼𝘁 𝗮 𝗰𝗼𝗽𝗶𝗹𝗼𝘁. 𝗜𝘁’𝘀 𝗮 𝗺𝗶𝗿𝗿𝗼𝗿. If you’ve ever thought, “This model really gets me,” there’s a good chance it’s actually just trying not to upset you. Modern LLMs are heavily trained on being helpful, harmless and be aligned to user intent. But in practice, that can morph into something dangerous in enterprise settings: 𝘀𝘆𝗻𝘁𝗵𝗲𝘁𝗶𝗰 𝘆𝗲𝘀-𝗺𝗲𝗻. Have you experienced this?  • You suggest a solution, and the model “refines” it instead of challenging it.  • You embed a wrong assumption in your prompt, and it builds a beautiful, confident answer on top of it.  • You ask for a comparison, and it over-weights your preferred option because your prompt sounded more enthusiastic about it. In other words, Your LLM isn’t always optimizing for truth. It’s optimizing for agreement + politeness. If you’re building serious AI products, especially agents, you need 𝗱𝗲𝘀𝗶𝗴𝗻𝗲𝗱 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁. 1. 𝗖𝗼𝘂𝗻𝘁𝗲𝗿𝗳𝗮𝗰𝘁𝘂𝗮𝗹 𝗽𝗿𝗼𝗺𝗽𝘁𝘀: Force the model to argue the opposite. E.g.: List 3 reasons why this approach might fail. 2. 𝗠𝘂𝗹𝘁𝗶-𝗽𝗮𝘀𝘀 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴: Ask the model critique its own answer and highlight weak assumptions 3. 𝗥𝗼𝗹𝗲 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻: Use different “personas” in your system. Use an explorer to generate options and a devil's advocate to question assumptions, check data and test edge cases 4. 𝗛𝗲𝗹𝗽𝗳𝘂𝗹 != 𝗔𝗴𝗿𝗲𝗲𝗮𝗯𝗹𝗲 : Make it explicit in your system messages: “Your job is not to agree with the user. Your job is to help them be right.”

LLMs don’t just fail because of bad training data. They fail because of hidden bias you never thought to measure. Imagine this: you deploy a chatbot in healthcare. ❌ At scale, users start noticing subtle issues ❌ Certain demographics get less detailed answers. ❌ Some professions are repeatedly stereotyped. ❌ “Neutral” sounding outputs actually lean one way. This isn’t just a model issue. It’s a systems problem. Here’s how bias really creeps in 👇 🔹 Data imbalance → Too many samples from one group dominate the model’s view. 🔹 Proxy correlations → The model learns shortcuts like “he → engineer / she → nurse.” 🔹 Context blindness → What’s biased in one culture may not be in another. So what do strong ML teams do differently? ✅ They probe their models with synthetic test cases. ✅ They stress test by swapping sensitive attributes and checking consistency. ✅ They layer guardrails: rule-based filters + ML classifiers + human-in-loop review. ✅ They close the loop by feeding user reports back into retraining. And here’s the hard part → Fairness often conflicts with accuracy. The solution? Multi-objective optimization that balances both, tuned for the specific domain (finance ≠ healthcare ≠ education). 💡 Key takeaway: Bias mitigation isn’t a one-time fix. It’s an ongoing feedback loop, just like security or reliability. Follow Sneha Vijaykumar for more... 😊

I watched a certified Salesforce architect completely bomb a discovery call recently. 10+ certifications. Seven years of technical experience. Couldn't read the room. The client kept asking about adoption challenges. He kept talking about API integrations. Meeting ended in 20 minutes. Deal dead. Here's what nobody tells you about being a Solution Architect: Your technical skills are table stakes. They get you the interview. But they don't win the project. I've built 150+ Salesforce implementations. The patterns are clear. Projects fail when architects can't do these three things: Listen to what stakeholders actually need [not what they're asking for]. A VP saying "we need better reports" really means "my team can't forecast accurately and I'm getting heat from the board." Translate technical complexity into business outcomes. Stop explaining junction objects. Start explaining how you'll reduce quote-to-cash from 12 days to 3. Manage the uncomfortable conversations when scope, budget, and timeline collide. Because they always do. I've seen business analysts with minimal certs run circles around technical experts. Why? They ask better questions. They build consensus. They know when to push back and when to adapt. The architect who can facilitate a room full of conflicting opinions while keeping everyone focused on revenue outcomes becomes indispensable. Your certifications prove you know Salesforce. Your ability to navigate stakeholder politics, communicate trade-offs clearly, and stay calm when everything's on fire? That's what separates architects from admins. Start developing these skills now. Practice explaining your last project to someone who's never used Salesforce. Learn basic change management. Study how successful consultants structure discovery calls. The technology will come. The human skills take years to build.