Engineering Workflow Management Systems

When working with multiple LLM providers, managing prompts, and handling complex data flows — structure isn't a luxury, it's a necessity. A well-organized architecture enables: → Collaboration between ML engineers and developers → Rapid experimentation with reproducibility → Consistent error handling, rate limiting, and logging → Clear separation of configuration (YAML) and logic (code) 𝗞𝗲𝘆 𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 𝗧𝗵𝗮𝘁 𝗗𝗿𝗶𝘃𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 It’s not just about folder layout — it’s how components interact and scale together: → Centralized configuration using YAML files → A dedicated prompt engineering module with templates and few-shot examples → Properly sandboxed model clients with standardized interfaces → Utilities for caching, observability, and structured logging → Modular handlers for managing API calls and workflows This setup can save teams countless hours in debugging, onboarding, and scaling real-world GenAI systems — whether you're building RAG pipelines, fine-tuning models, or developing agent-based architectures. → What’s your go-to project structure when working with LLMs or Generative AI systems? Let’s share ideas and learn from each other.

Monthly Book Review: Two reads for building real AI systems (from architecture to agents) 📘 𝗟𝗟𝗠𝘀 𝗶𝗻 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 (conceptual, system-level view) In short, this one is about ‘how to think about LLMs in business systems’. It focuses on how LLMs are deployed and integrated into organizations - covering architecture, governance, scaling, evaluation, and real-world adoption patterns. I’d say it’s especially useful for shaping the mindset around frameworks and understanding how LLMs actually fit into enterprise infrastructure. 𝗪𝗵𝗮𝘁 𝘀𝘁𝗼𝗼𝗱 𝗼𝘂𝘁 𝘁𝗼 𝗺𝗲: -  Clear breakdowns of common architecture patterns (RAG, fine-tuning, deployment, governance, etc.) - Strong focus on integration with existing workflows and data systems - Practical discussion of risk, cost, and compliance trade-offs 𝗕𝗲𝘀𝘁 𝗳𝗼𝗿 (𝗶𝗺𝗼): ▪️Technical leads moving into architecture or management roles ▪️Engineers and managers who want to understand the full picture ▪️Non-technical leaders looking to understand how LLMs can fit into their current stack 📙 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 𝗶𝗻 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲 (hands-on and builder-focused) This one’s much more practical and tutorial-style. You’ll learn how to build agentic systems that connect to tools, APIs, and external data sources. 𝗪𝗵𝗮𝘁 𝗶𝘁 𝗰𝗼𝘃𝗲𝗿𝘀: ➤ Step-by-step use of LangChain, LlamaIndex, and similar frameworks ➤ Multi-agent workflows, reasoning loops, and task execution ➤ Code examples that bring together planning, memory, and real-world orchestration 𝗠𝘆 𝘁𝗮𝗸𝗲: If you’re building anything agentic, this is a great one to keep on your desk. It does assume you’re already comfortable with ML foundations and some coding, but nothing very advanced. ***Both books are great, but serve different needs. You don’t need to read them in order, but if you plan to go through both, I’d start with LLMs in Enterprise and follow with AI Agents in Practice. It’s a natural flow from systems to agents. Hope this helps anyone exploring this space, would love to hear if you’ve read either, or if you’ve got others to recommend. 🔗Links to both books below (both first edition): ✔️ AI Agents in Practice by Valentina Alto https://packt.link/RIVbG ✔️ LLMs in Enterprise by Ahmed Menshawy and Mahmoud Fahmy https://packt.link/wu2d7 __________ For more on AI and learning materials, plz check my previous posts. I share my journey here. Join me and let's grow together. Alex Wang #aiagents #agenticai #enterpriseai #bussiness

Power Automate Work Queues are not built for scale! That's a fact. When you think about scalability in Power Automate, one thing that will definitely come to mind at some point is queues and workload management. While you might be able to survive without them in some event-based transactional flows that only process a single item at a time, but whenever you process tasks in batches, or when RPA gets involved, you'll need queues. Power Automate comes with Work Queues out of the box. And you would think that's your go-to queueing mechanism for scaling. After all, it's at scale that you really need those queues - to de-couple your flows and make it easier to maintain, support, debug them, as well as make them more robust and efficient. Queues is a must even at medium scale. Heck, we use them even in small scale implementations. But the surprising thing about Power Automate Work Queues is that they are not fit for high scale implementations. And that is by design! The docs themselves (link in the comments) explicitly state that if have high volumes or if you dequeue (pick up work items from the queue for processing) concurrently, you should either do it within moderate levels or use something else. If you try and use Power Automate Work Queues for high scale implementations (more than 5 concurrent dequeue operations or hundreds/thousands of any type operations involving the queues), you'll get in trouble. There can be all sorts of issues that could happen - your data may get duplicated, you may accidentally deque the same work item in multiple concurrent instances, or your flows might simply get throttled or even crash. This is because of the way they're build and the way they utilize Dataverse tables for storing work items and work queue metadata. So, if you do want to scale, it's best to use an alternative. And, obviously, Microsoft wouldn't be Microsoft if they didn't have an alternative tool to do that. The docs themselves recommend Azure Service Bus Queues for high throughput queueing mechanisms. Another alternative could also be Azure Storage Queues, but that only makes sense if the individual work items in your queue can get large (lots of data or even documents) or when you expect your queue to grow beyond 80GB (which is possible in very large scale implementations). Otherwise, Azure Service Bus Queues are absolutely perfect for very large volumes of small transactions. On top of that, they have some very advanced features for managing, tracking, auditing and otherwise handling your work items. And, of course, there's a existing connector in Power Automate to use it. So, while I do love Power Automate Work Queues, I'll only use them in relatively small scale implementations. And for everything else - my queues will go to Azure. And so should yours.

Enterprise AI teams are struggle with agents that cannot reliably access and act on core enterprise systems and data, lack deep understanding of domain‑specific language and context, and sit on top of weak retrieval and grounding over internal knowledge, which leads to brittle answers and hallucinations. At the same time, organizations have poor observability, control and evaluation of AI behaviour, with limited guardrails, benchmarking and testing and they find it hard to compose multiple agents and tools into robust workflows, so initiatives stall in “pilot purgatory” instead of scaling into production. Part of the solution to these challenges is integrating domain‑aware models with enterprise systems and adding strong retrieval, monitoring, guardrails and orchestration for multi‑agent workflows.

As an advisor to tech scaleups, and a former CTO and SVP of Engineering,  I've often encountered a familiar CEO complaint: "Our engineering team is too slow!" However, focusing solely on increasing individual productivity is rarely the solution. Sometimes the answer is changing the organizational structure. 🔍 The Issue with Flat Structures: Time to market was a major problem in a scale-up I advised, even though they had a flat structure where 40+ engineers reported directly to the VP of engineering and all of them shared equal accountability to the delivery of the software. 🚧 The Consequences: Major overcommitment.  People raised their hands to take on work even if the group was super extended. There was nobody that fully understood the team’s capacity vs the actual workload they took on. This approach led to a lack of predictability, chronic delays, unhappy customers, and ultimately, a tarnished reputation. 🛠️ The Solution: Transitioning to a hierarchical structure with focused teams and accountable experienced leaders was the game-changer. This shift brought in clarity, accountability, and much-needed structure. 📈 The Results: Predictable schedules, improved customer satisfaction, and a thriving engineering culture. ✅ Takeaways for Your Organization: Examine your organization with critical eyes: Is your ownership and accountability structure clear? Are your teams sized and focused appropriately? Do your leaders have the authority to deliver effectively? For more on the case study and about building a sustainable, efficient, and customer-centric engineering team in the blog post. 💭 I'm curious to hear your thoughts: Have you faced similar challenges? How did you address them? Let's share insights and grow together! #EngineeringManagement #Leadership #Productivity  _______________ ➡️ I am Talila Millman, a fractional CTO,  a management advisor, and a leadership coach. I help CEOs and their C-suite grow profit and scale through optimal Product portfolio and an operating system for Product Management and Engineering excellence.  📘 My book The TRIUMPH Framework: 7 Steps to Leading Organizational Transformation will be published in Spring 2024 https://lnkd.in/eVYGkz-e

Ever wonder why we tend to solve problems the hard way? 🤔 The key is in how we connect the dots. A cancer hospital was facing a major challenge. Patients, often anxious, needed timely care without added delays. Doctors relied on quick access to medical images to make this possible. For most hospitals, loading images within three seconds is the standard. But cancer patients often have extensive imaging records, making this target a significant challenge. This created escalating pressure in an environment that's already stretched to its limits The hospital consulted several firms. They all suggested the same thing: a costly network upgrade that would disrupt daily operations and inconvenience patients even more. The proposed solution was out of the question, the hospital needed something affordable that wouldn’t disrupt patient care. A consulting firm graciously recommended me for the task. I saw the problem from a different angle. IT experts looked at the network. But as a Health Informatician, I focus on using data and technology to design health services that support optimal care delivery. Instead of waiting for doctors to request images, why not load them in advance? By preparing the images during the patient’s wait time, we created a seamless workflow without costly upgrades. The results were immediate and impactful. 😊 The hospital easily met the three-second target, and patients noticed the improvement with shorter wait times. The cost savings were substantial, all without any disruption to care. "Adam, you literally performed magic!” shared the hospital’s clinical operations lead. Sometimes, the simplest solutions make the biggest difference. The key was understanding how health services connect and using technology to support these connections. These days, as a digital health transformation coach, I continue to co-design sustainable, human-centered innovations that improve how information is used to advance health outcomes. Ever found a simple solution to a complex challenge? I’d love to hear your insights and share approaches that make an impact. #HealthcareInnovation #LeadershipLessons #DigitalTransformation

One of the biggest challenges I see with scaling LLM agents isn’t the model itself. It’s context. Agents break down not because they “can’t think” but because they lose track of what’s happened, what’s been decided, and why. Here’s the pattern I notice: 👉 For short tasks, things work fine. The agent remembers the conversation so far, does its subtasks, and pulls everything together reliably. 👉 But the moment the task gets longer, the context window fills up, and the agent starts forgetting key decisions. That’s when results become inconsistent, and trust breaks down. That’s where Context Engineering comes in. 🔑 Principle 1: Share Full Context, Not Just Results Reliability starts with transparency. If an agent only shares the final outputs of subtasks, the decision-making trail is lost. That makes it impossible to debug or reproduce. You need the full trace, not just the answer. 🔑 Principle 2: Every Action Is an Implicit Decision Every step in a workflow isn’t just “doing the work”, it’s making a decision. And if those decisions conflict because context was lost along the way, you end up with unreliable results. ✨ The Solution to this is "Engineer Smarter Context" It’s not about dumping more history into the next step. It’s about carrying forward the right pieces of context: → Summarize the messy details into something digestible. → Keep the key decisions and turning points visible. → Drop the noise that doesn’t matter. When you do this well, agents can finally handle longer, more complex workflows without falling apart. Reliability doesn’t come from bigger context windows. It comes from smarter context windows. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗔𝗜 𝗜𝘀 𝗮 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 Much of today’s conversation around AI agents focuses on #graphs, #models, #prompts, #context, or orchestration #frameworks. These topics matter, but they rarely determine whether an AI system succeeds once it moves from prototype to enterprise production. The real challenges appear when AI systems operate inside long-running business workflows. Consider a workflow that analyzes documents, retrieves data from multiple systems, calls APIs, and produces a structured decision. Such processes may run for twenty or thirty minutes and involve dozens of steps. Now imagine something routine happens: a network call fails, an API times out, or a container restarts. No problem, the agent says. It starts the workflow again. That may be acceptable for chatbots. It quickly becomes impractical for enterprise processes such as financial analysis, document processing, underwriting, or claims review. These workflows are long-running, resource-intensive, and deeply connected to operational systems. In these situations, the limitation is rarely the model’s intelligence. More often, the challenge lies in the #engineering #discipline around the system. At Cognida.ai, our focus is on building practical enterprise AI systems rather than demos or PoCs. We consistently find that several principles from #distributedsystems engineering become essential once AI moves into production. Here are three such constructs: 𝗗𝘂𝗿𝗮𝗯𝗹𝗲 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 Agent workflows should not be treated as temporary requests. Each step should persist its state so that if a failure occurs, the system can resume from the last successful step rather than restarting the entire process. In practice, this means workflow orchestration with checkpointed state, deterministic execution, and event-driven recovery. For long-running processes, this is often the difference between a prototype and a production system. 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝘁 𝗔𝗰𝘁𝗶𝗼𝗻𝘀 AI agents increasingly trigger real-world actions: sending emails, calling APIs, updating records, moving files, or initiating financial transactions. Retries are inevitable in distributed systems. If actions are not idempotent, retries can create duplicate or inconsistent results. Reliable AI systems must ensure the same action cannot run twice unintentionally. 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗦𝘁𝗮𝘁𝗲 𝗕𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗠𝗼𝗱𝗲𝗹 Large language models operate within limited context windows rather than durable memory. Enterprise workflows often run longer and across many stages. The system managing the workflow must maintain its own persistent state instead of relying on the model’s temporary context. It means treating AI workflows as structured state machines, not simple prompt-response interactions. Are you treating AI workflows more like state machines, event-driven systems, or traditional #microservices? #PracticalAI #EnterpriseAI

People who’ve used Airflow long enough are familiar with these common pain points: rigid DAGs, unclear observability, and frustrating scaling. Debugging a failing workflow at scale feels like archaeology, digging through logs, piecing together what went wrong. I’m guessing this is why I’m hearing about data operators moving to Prefect. The reason? It’s Python-native, flexible, and removes the operational overhead of managing an Airflow cluster. More importantly, it treats orchestration as a first-class concern, not an afterthought. I checked out a few of their blogs, and here’s what stands out: ✅ Their event-driven workflows: You can now move beyond static schedules. Prefect reacts to real-time events, making it ideal for dynamic pipelines. ✅ They enabled decoupled scheduling: No more Airflow’s single scheduler bottleneck. Prefect lets teams deploy and scale workflows independently. ✅ They have built-in observability: No more wondering where a task failed. Prefect provides full visibility without extra plugins. ✅ The dynamic infrastructure model: scale your infrastructure based on the specific workflow’s need with work pools. If Airflow feels like overhead, Prefect might be your next move. I would love to know who made the switch, as I am curious to hear your experience. Check out the blog posts below! #data #workflows #datapipelines

Small variations in prompts can lead to very different LLM responses. Research that measures LLM prompt sensitivity uncovers what matters, and the strategies to get the best outcomes. A new framework for prompt sensitivity, ProSA, shows that response robustness increases with factors including higher model confidence, few-shot examples, and larger model size. Some strategies you should consider given these findings: 💡 Understand Prompt Sensitivity and Test Variability: LLMs can produce different responses with minor rephrasings of the same prompt. Testing multiple prompt versions is essential, as even small wording adjustments can significantly impact the outcome. Organizations may benefit from creating a library of proven prompts, noting which styles perform best for different types of queries. 🧩 Integrate Few-Shot Examples for Consistency: Including few-shot examples (demonstrative samples within prompts) enhances the stability of responses, especially in larger models. For complex or high-priority tasks, adding a few-shot structure can reduce prompt sensitivity. Standardizing few-shot examples in key prompts across the organization helps ensure consistent output. 🧠 Match Prompt Style to Task Complexity: Different tasks benefit from different prompt strategies. Knowledge-based tasks like basic Q&A are generally less sensitive to prompt variations than complex, reasoning-heavy tasks, such as coding or creative requests. For these complex tasks, using structured, example-rich prompts can improve response reliability. 📈 Use Decoding Confidence as a Quality Check: High decoding confidence—the model’s level of certainty in its responses—indicates robustness against prompt variations. Organizations can track confidence scores to flag low-confidence responses and identify prompts that might need adjustment, enhancing the overall quality of outputs. 📜 Standardize Prompt Templates for Reliability: Simple, standardized templates reduce prompt sensitivity across users and tasks. For frequent or critical applications, well-designed, straightforward prompt templates minimize variability in responses. Organizations should consider a “best-practices” prompt set that can be shared across teams to ensure reliable outcomes. 🔄 Regularly Review and Optimize Prompts: As LLMs evolve, so may prompt performance. Routine prompt evaluations help organizations adapt to model changes and maintain high-quality, reliable responses over time. Regularly revisiting and refining key prompts ensures they stay aligned with the latest LLM behavior. Link to paper in comments.