Evaluating the Effectiveness of Automated Workflows

There’s a huge difference between ‘I got AI to do this amazing thing for social media points’ and ‘I got AI to do this thing that generates a lot of revenue for my business or our clients.’ Real-world AI is very different. Most agents require small language models. Large context windows and multiple rounds of model calls turn the unit economics of foundational models negative for many use cases. Everything we build for clients starts with local AI. We spend no more than 2 days trying to get the workflow running on the Dell Pro Max T2 in my office. If it won’t run locally, using a frontier model rarely changes that. We scale the agent to support a small set of early adopters. This phase is critical. An early adopter cohort has been trained to use agents at their earliest maturity phase. Most users would reject the agent in this raw form. But this phase is intended to rapidly improve the agent’s workflow integration, orchestration, and reliability. Human feedback from trained early adopters improves agent performance faster than any other approach I have found. We iterate on more than just the LLMs. This phase fills in the knowledge graph, improves tool usage, adds guardrails, and informs the usage of more traditional machine learning models to augment the agent. When improvements plateau, we assess the agent. It is only promoted if its impact on outcomes meets user or customer expectations. Is it valuable? How does it reorchestrate workflows? Can the business monetize it? We roll the agent out to an alpha release cohort to scale the feedback flywheel. At this point, we know we have something valuable. We’re trying to improve its reliability and handle more workflow variations before a wider launch. We only evaluate frontier model usage at this phase. We finally know enough to make targeted decisions about where in the workflow frontier model performance could make a big enough difference to be worth considering. The alpha release also reveals adoption barriers for the agent and reorchestrated workflow. Most agents require us to craft an adoption journey for users and customers. That typically includes training for internal users and a phased rollout for customers. When improvement plateaus again, the agent is ready for general release. The process takes 2-3 months, and only about 30% of the workflows we try in my office end up going the distance. Data and information architecture make a huge difference. One client with a very mature knowledge graph is seeing a workflow success rate of over 50%. Small models perform significantly better for their use cases. #DellProMax

The AI workflow produced great results, yet people did not feel safe relying on the output. ⛔ That was the situation I encountered in a client workshop in Brussels last week, and it is far more common than most organisations like to admit. The team had invested time and effort into designing an AI-supported workflow. The use case was clear, the technical setup was sound, the data quality was acceptable, and the people involved had already received training on how to use AI. Despite all of this, the workflow was barely used in practice. People ran the AI step, reviewed the output, and then quietly redid the work themselves. During the workshop, we mapped the real workflow together, step by step, focusing not on how the process was documented but on how the work actually happened on a normal working day. At one point, a participant looked at the whiteboard and said: “I only trust the result after I have checked it myself anyway.” That sentence shifted the entire conversation. As we continued mapping the process, a pattern became visible: Everyone validated AI outputs differently.  Some checked everything, even low-risk drafts.  Others barely checked high-risk decisions. Accountability was assumed but never explicitly defined. Human validation was happening constantly, but it was invisible, inconsistent, and highly personal. We redesigned the workflow and introduced a simple checklist for built-in human validation. 💡 This checklist replaced individual safety habits with a shared, explicit process. ✅ Define the risk level of the output. Clarify whether the AI output is a draft, a recommendation, or a decision with external impact. ✅ Decide if validation is required. Make it explicit which outputs require human review and which can flow through without intervention. ✅ Specify the validation moment. Define when validation happens in the workflow and before which downstream step. ✅ Assign clear responsibility. Name the role that validates the output and the role that makes the final decision. ✅ Separate generation from judgment. Ensure the AI prepares content or options, while humans remain accountable for approval and outcomes. ✅ Remove unnecessary checks. Regularly review the workflow to eliminate validation steps that add friction without reducing risk. Once this checklist was applied, people felt much more confident about the AI output because they knew when human judgment was required. 👉 Is human validation in your AI workflows clearly designed, or is it still improvised? Let’s discuss.

Laboratory automation is often judged on its price tag before its true value is realized. When organizations evaluate automation projects, the first questions tend to be: What does it cost? What’s the ROI? Those are valid business concerns, but focusing only on financials overlooks a critical driver of value: quality of results. Automation consistently reduces human error, enforces standardized workflows, and provides reproducibility at a scale that manual processes simply cannot match. For biobanks, drug discovery, and clinical research alike, this translates into: • More reliable data • Fewer repeat experiments • Accelerated timelines Of course, automation requires upfront investment: capital equipment, integration, training, and change management. But when you factor in not only efficiency gains but also the cost of poor quality (failed assays, sample loss, compromised reproducibility), the ROI calculation looks very different. In other words: the business case for automation is not just about doing work faster or cheaper - it’s about doing it right, every time. And that reliability is where the real value lies. I’m curious - when your organization makes automation decisions, do you prioritize ROI first, or quality first? #LRIG #LabAutomation #LaboratoryAutomation #DrugDiscovery #Biobanking #LaboratoryRobotics

This week I automated the process of identifying which clients are wrapping up a training and do not have anything else scheduled with us afterward. This week I built a small Power Automate flow that solves a problem we kept bumping into, but never took the time to automate. We store all of our client trainings in a single SharePoint list. Past, present, and future sessions all live together. The data was there, but the insight was not. The question we wanted to answer was simple: → Which clients are finishing a training this month and do not have anything else scheduled with us afterward? Manually, that meant filtering dates, scanning company names, cross checking future sessions, and then writing a follow up email. It worked, but it never happened as consistently as it should. So I automated it. Here is what the flow does: 1. First, it runs automatically on the first of every month. 2. It pulls all trainings that occur during the current month from SharePoint. 3. From there, it evaluates each company on that list and checks whether they have any trainings scheduled after the current month. If they do, the flow ignores them. 4. If they do not, the automation captures the company name and the name of their most recent training session and formats the results into a clean bulleted list. 5. Finally, it sends an email to our Director of Client Services with that list included in the body. Each bullet shows the company name and their latest training so follow up conversations are grounded in context. The email also includes a link to our full training library so she can easily dig deeper if needed. The outcome is simple but powerful. ★ Leadership gets a proactive view of clients who may need follow up. ★ Client services can prioritize outreach without pulling reports. No one has to remember to run a manual check every month. This is a good example of how automation does not need to be flashy to be valuable. Sometimes the best flows just make sure the right information reaches the right person at the right time, every time. If you are sitting on good data but still relying on reminders and manual checks, this is usually a sign there is an automation opportunity waiting. Let’s start building!

A tiny 6-person company hired me to help them optimize their ops. Here’s exactly what I did to unlock $50k in profit: 📈 Context: → Small team of 6 people, growing fast but bleeding money in operations → Owner working 30 hours/week, mix of full-time and part-time contractors → Processes were completely manual → everything took forever → They knew they had inefficiencies but had no idea where to start After the initial call, I knew there were tons of things to improve. So here’s what I did to audit their situation: 1️⃣ Calculate your team's TRUE hourly rate → Owner: 30 hrs/week × $100/hr = $3,000 → Team members: Various rates from $20-30/hr  → Total weekly cost: $5,000 across 114 hours  → Effective rate: $44/hour (this number is crucial) 2️⃣ Map every business process → Cold email tracking → Client onboarding workflows → Content creation & review → Project tracking across multiple tools → Weekly newsletter creation → Deliverable handoffs 3️⃣ Score each opportunity on 3 factors → Time savings: How many hours/week would this save?  → Implementation complexity: Simple (10) vs Complex (1) → Business impact: Revenue impact vs nice-to-have For example, this is what it looked like with an automated email follow-up: – Time savings: 5 hours/week – Implementation: 5 (must be reliable if client-facing) – Business impact: 10 (directly drives client results) – Total score: 20 🔥 The result → 30 hours/week in time savings  → 30 hours × $44/hour = $1,320 weekly savings  → Annual impact: $68,640 in recovered time The best part? Besides money and time, it made EVERYTHING feel much better. Clients are happier. The team is way less stressed. And the owner can finally focus on working ON the business, not in. How much time do you think you could save if you’d automate a few bits of your business? Let’s chat. Here’s my calendar for a quick intro call: https://cal.link/ftmintro

𝐓𝐡𝐞 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐯𝐬. 𝐄𝐟𝐟𝐞𝐜𝐭𝐢𝐯𝐞𝐧𝐞𝐬𝐬 𝐀𝐮𝐝𝐢𝐭 Run this once a month. For every task you used AI for, answer four questions: → Did this save time or change a decision? → Was the output tied to a strategic priority? → Did it improve an outcome or reduce effort? → Would a stakeholder care about the result? Mostly "saved time" and "reduced effort"? You are using AI efficiently. Mostly "changed a decision" and "improved an outcome"? You are using AI effectively. The goal is not to stop using AI for efficiency. It is to make sure effectiveness is also in the mix.

When I first started exploring automation, I had one big question: How do I know if a process is truly ready for automation? The key lies in identifying the right processes that will bring the most value when automated. Here’s a breakdown of what to look for: - Repetitive & Rule-Based Tasks: Processes that are repetitive, time-consuming, and prone to human error are prime candidates for automation. - High Frequency of Execution: The more often a process is performed, the more significant the benefits of automation. - Standardized & Well-Defined Steps: When a process has clear, standardized steps, it’s easier to automate. - Data-Driven: Processes involving large volumes of data or consistent data processing are ideal for automation. - Manual & Time-Consuming: Automating processes that require significant manual effort saves time and resources. - Clear Start & End Points: Processes with well-defined triggers and outcomes are easier to automate. - Measurable Outcomes: When results are quantifiable, it’s easier to track the impact and ROI of automation. - High Impact: Focus on processes that will drive efficiency, cost savings, or customer satisfaction. - Low Complexity: Start with simpler processes for quicker wins and smoother implementation. - Few Exceptions: Processes that follow a consistent path with minimal variations are easier to automate effectively. A useful framework for assessing readiness is the R.E.A.D.Y. approach: - Repeatable: The process is performed regularly. - Executable: The steps are clear and actionable. - Auditable: The results are trackable and measurable. - Dependent: The process relies on specific inputs and produces defined outputs. - Yield: The process will deliver significant ROI when automated. By using this approach, organizations can identify the best candidates for automation and prioritize their efforts for maximum impact. What processes in your organization meet these criteria? It might be time to take them to the next level with automation. #Automation #Efficiency #DigitalTransformation

Time for your weekend long-read! I just published a piece on something I think every organization using AI needs to understand: how to evaluate whether your AI systems actually work for your specific needs, both for legal tech and generally for business. Why? Generic benchmarks tell you how AI performs in abstract scenarios, but they miss your edge cases, your terminology, your standards. The gap between benchmark scores and real performance isn't just numbers - it's damaged trust and sleepless nights. Therefore, what is needed are systematic evaluations of YOUR specific products, workflows, and other applications of AI.  In the post I make the case that articulating quality AI outputs for in a way that can be objectively evaluated is now a core function of executive leadership and governance. Building custom evaluations isn't as technical as you might think. If you can articulate what good work looks like to a human employee, you can create meaningful AI evaluations. I know this first hand because much of my own consulting business service has transformed from help creating or improving existing applications to evaluations of applications to ensure they are hitting quality thresholds and staying under cost and risk ceilings. Custom evals are essential for successful application of AI and they are the key (frequently missing) method and mechanism for AI governance. I've also released Lake Merritt, an open-source platform that makes this accessible. The quick start guide walks you through simple exercises you can try in just a few minutes to see how this works and to become directly familiar with evals. Lake Merritt is designed for business, legal, and other non-technical leaders to be able to quickly get involved with evals. While the software is still in early public beta, in the context of the blog post you can use Lake Merritt to understand and get started with evals and even to get a start on your own internal evals. It also supports more advanced workflows such as OpenTelemetry and multi-agent system evals. The aspect I like best is how we use "eval packs" so you can version and even share your approaches to evals with other in your organization or swap them with the broader community as we all learn together what the best methods are.  More on Lake Merritt here: https://lnkd.in/g2Q7CA2c Many thanks to Artificial Lawyer for noting the launch of Lake Merritt earlier this week in their fine article, here: https://lnkd.in/g55zPbpt Likewise, in the blog post I took a moment to recognize some of the folks in AI evals who I think you should also be paying attention to, including Vals AI, Arize AI, Galileo, Anna Guo, Darius Emrani, and many others! Would love to hear your thoughts on how evaluation fits into your AI strategy, or if you've wrestled with the challenge of measuring AI quality in your own context. Link: https://lnkd.in/gKqMmjQw

Most teams buy AI agents like they buy software. Plug it in. Expect ROI. Then spend weeks cleaning up the output. I've watched marketing teams throw agents at "content creation" and "campaign launches" without ever mapping what those workflows actually look like. The result? Agents running in circles. Humans cleaning up messes. Leadership asking why the expensive AI isn't delivering ROI. The fact is if the workflow is invisible, the agent guesses. Execution collapses. Here's what I mean: 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝟭: 𝗖𝗼𝗻𝘁𝗲𝗻𝘁 𝗖𝗿𝗲𝗮𝘁𝗶𝗼𝗻 Most teams say: "We want AI to create content." That's not a workflow. That's a wish. A workflow looks like this: 𝗦𝘁𝗲𝗽 𝟭: 𝗧𝗼𝗽𝗶𝗰 𝗜𝗱𝗲𝗻𝘁𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 → Input: Content calendar, trending topics, audience questions → Output: Prioritized topic with angle and target audience → Human checkpoint: Approve topic before proceeding 𝗦𝘁𝗲𝗽 𝟮: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 & 𝗢𝘂𝘁𝗹𝗶𝗻𝗲 → Input: Approved topic + brand guidelines + competitor content → Output: Structured outline with key points and sources → Human checkpoint: Review outline for strategic alignment 𝗦𝘁𝗲𝗽 𝟯: 𝗙𝗶𝗿𝘀𝘁 𝗗𝗿𝗮𝗳𝘁 → Input: Approved outline + voice pack + example posts → Output: Complete draft matching brand voice → Human checkpoint: Edit for accuracy and tone 𝗦𝘁𝗲𝗽 𝟰: 𝗩𝗶𝘀𝘂𝗮𝗹 𝗔𝘀𝘀𝗲𝘁𝘀 → Input: Final copy + brand templates → Output: Formatted graphics, carousel, or video brief → Human checkpoint: Approve visuals 𝗦𝘁𝗲𝗽 𝟱: 𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 → Input: Final content + channel specs + scheduling parameters → Output: Scheduled posts across platforms → Human checkpoint: Final review before publish Without this map, an agent doesn't know: → Where to start → What inputs it needs → When to pause for human review → What "done" looks like 💡 Reality: "Create content" isn't a workflow. It's five workflows stitched together with decision points. 𝗧𝗵𝗲 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗠𝗮𝗽𝗽𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 Before you deploy any agent, answer these questions for each workflow: → What triggers this workflow? → What are the discrete steps? → What inputs does each step require? → What outputs does each step produce? → Where do humans need to review or approve? → What does "done" look like? → How do we measure success? Save this for your next AI planning session.