How to Implement Iterative Improvement in AI Strategies

What goes around comes around... and it turns out a quality management principle from the 1950s might be the most important thing missing from how most people use AI coding agents today. W. Edwards Deming transformed Japanese manufacturing with one simple idea: Plan-Do-Check-Act. A continuous cycle where every iteration makes you better than the last. Not a straight line. A spiral. What you invest in understanding comes back as wisdom. What you skip comes back as repeated failure. I've been using AI agents like Claude Code for a while now, and honestly the biggest productivity gains don't come from the AI writing code faster. They come from wrapping the AI in a proper improvement cycle. Here's what I mean: 1. **Plan** - Before writing a single line of code, use your agent in planning mode. Ask clarifying questions. Map out the approach. Get alignment before you start. Deming said "If you can't describe what you are doing as a process, you don't know what you're doing." That applies to prompting AI just as much as it does to manufacturing. 2. **Do** - Execute the plan. Let the agent build, but maintain a living document of what's being done, what's next and what's blocked. Track the work as it happens, not after. 3. **Check** - This is where I think many have stopped. They ship the AI output without validating it. Write tests. Review the output critically. Compare what you expected against what you got. Deming preferred to call this "Study" because checking isn't enough... you need to actually understand the gap between prediction and reality. 4. **Act** - Here's the part almost nobody does. Capture what you learned. Maintain a lessons file. What worked? What didn't? What should the agent do differently next time? This is where the cycle pays dividends because those lessons feed directly into the next Plan phase. I've started baking this into how I set up every AI coding project. My agent instructions literally follow PDCA... plan first and get confirmation, maintain a project tracker during execution, create tests to validate output, and keep a running lessons learned file that improves the process continuously. The result? Each cycle gets tighter. The agent gets better context. The output gets more reliable. The spiral goes up, not around in circles. Deming also said "A bad system will beat a good person every time." I'd add: a bad process will beat a good AI model every time. GPT-4, Claude, Gemini... none of it matters if you're just throwing prompts at a wall and hoping for magic. The teams that will win with AI agents aren't the ones with the best models. They're the ones with the best cycles around those models. What goes around, comes around. Deming knew it 70 years ago. It's just as true now... thanks John Willis! Are you running your AI coding workflows as a cycle, or just firing and forgetting? What does your improvement loop look like? #AI #PDCA #Deming #ContinuousImprovement #ClaudeCode #SoftwareEngineering #AIAgents

I think AI agentic workflows will drive massive AI progress this year — perhaps even more than the next generation of foundation models. This is an important trend, and I urge everyone who works in AI to pay attention to it. Today, we mostly use LLMs in zero-shot mode, prompting a model to generate final output token by token without revising its work. This is akin to asking someone to compose an essay from start to finish, typing straight through with no backspacing allowed, and expecting a high-quality result. Despite the difficulty, LLMs do amazingly well at this task! With an agentic workflow, however, we can ask the LLM to iterate over a document many times. For example, it might take a sequence of steps such as: - Plan an outline. - Decide what, if any, web searches are needed to gather more information. - Write a first draft. - Read over the first draft to spot unjustified arguments or extraneous information. - Revise the draft taking into account any weaknesses spotted. - And so on. This iterative process is critical for most human writers to write good text. With AI, such an iterative workflow yields much better results than writing in a single pass. Devin’s splashy demo recently received a lot of social media buzz. My team has been closely following the evolution of AI that writes code. We analyzed results from a number of research teams, focusing on an algorithm’s ability to do well on the widely used HumanEval coding benchmark. You can see our findings in the diagram below. GPT-3.5 (zero shot) was 48.1% correct. GPT-4 (zero shot) does better at 67.0%. However, the improvement from GPT-3.5 to GPT-4 is dwarfed by incorporating an iterative agent workflow. Indeed, wrapped in an agent loop, GPT-3.5 achieves up to 95.1%. Open source agent tools and the academic literature on agents are proliferating, making this an exciting time but also a confusing one. To help put this work into perspective, I’d like to share a framework for categorizing design patterns for building agents. My team AI Fund is successfully using these patterns in many applications, and I hope you find them useful. - Reflection: The LLM examines its own work to come up with ways to improve it. - Tool use: The LLM is given tools such as web search, code execution, or any other function to help it gather information, take action, or process data. - Planning: The LLM comes up with, and executes, a multistep plan to achieve a goal (for example, writing an outline for an essay, then doing online research, then writing a draft, and so on). - Multi-agent collaboration: More than one AI agent work together, splitting up tasks and discussing and debating ideas, to come up with better solutions than a single agent would. I’ll elaborate on these design patterns and offer suggested readings for each next week. [Original text: https://lnkd.in/gSFBby4q ]

It’s no secret that consultants charge more to clean up a mess than to do it right the first time. However, with AI products, mistakes and iterative improvement are part of the process. The key is to prevent mistakes from becoming a mess. People often call iterative delivery slow, but it’s actually faster than the alternative. Iterative delivery keeps mistakes small. Continuous feedback from each delivery makes critical updates and high-value improvements obvious. If it takes a year to deliver an AI product, a year’s worth of assumptions are built into it. Real-world feedback feels like a firehose. It’s hard to determine which assumptions were flawed and what to do about them. That’s how mistakes become a mess. Some mistakes are self-inflicted but avoidable. Just doing something with AI or taking an “if we build it, they will buy” approach ends in a mess. Opportunities must lead technology, and core strategy must remain in control. Chasing AI hype is guaranteed to make a mess. Building a vision based on product demos and webinars leads to an expensive AI cleanup. The wrong vision is worse than no vision when sunk costs prevent the business from changing tracks early. Unless you’re on the technical side, steer clear of vibe coding and no-code solutions that make outrageous claims. It’s good to experiment with new solutions, tools, and methods, but don’t bet the farm on them until they prove themselves. It’s critical to experiment with AI pragmatically. Take intelligent risks and use feedback to stay grounded in reality. Use the roadmap to maintain alignment with a holistic vision for the AI platform’s maturity rather than releasing disconnected one-off features.

Stanford University researchers share a model (with code) that iteratively boosts multi-agent performance on tasks like reasoning and negotiation by up to 21%, learning based on past interactions, calling it SiriuS as an acryonym for Self-improving Multi-agent Systems. A number of others are applying similar approaches. Multi-agent systems are both intrinsically complex, so difficult to configure, but also particularly amenable to iterative optimization, since data on individual agent actions as well as system performance are readily available. Key insights from the paper (link in comments) include: 📚 Experience libraries turn past mistakes into training data. Instead of relying on manually designed prompts, SiriuS builds a repository of successful reasoning steps while refining failed ones. This allows agents to learn without direct supervision, making multi-agent systems more adaptive and efficient over time. 🔄 Augmenting failed trajectories strengthens AI learning. When an agent makes a mistake, SiriuS doesn’t discard the attempt—it modifies and regenerates the response with feedback from another agent. This iterative correction process significantly boosts problem-solving accuracy in fields like biomedical QA and physics problem-solving. 🎭 Role specialization in multi-agent AI enhances performance. By assigning specific expertise to agents (e.g., physicist, mathematician, summarizer), SiriuS maximizes efficiency in solving complex problems. This structured division of labor enables a coordinated, systematic approach to AI problem-solving. 💬 Negotiation and competition are improved with self-optimization. SiriuS-trained agents perform better in economic simulations like resource exchanges, seller-buyer pricing, and ultimatum games. They achieve higher win rates and better payoffs, proving that AI can learn effective competitive and cooperative strategies autonomously. ⚖️ Actor-Critic frameworks refine AI judgment and correction. Using a critic agent to provide feedback and a judgment agent to validate solutions, SiriuS ensures that incorrect responses are properly identified and fixed. This method significantly improves reasoning accuracy compared to standard self-correction methods. Scalability of multi-agent performance is critical. This is a promising architecture. More coming on paths to improved agentic AI performance.

Rare moment... OpenAI’s new “From Experiments to Deployments” guide actually reflects how AI gets built and deployed in the real world. Their four-phase approach... foundations, fluency, prioritization, then build and scale, tracks almost exactly with what many of us have been talking about… what it actually takes to move AI from experimentation to production at scale. It all starts with the foundation. 1️⃣ The guide opens with something that can’t be emphasized enough… executive alignment, governance designed for speed, and data access come before the flashy deployments. Not as a blocker, but as the enabler for everything else. They’re explicit about this. Companies that skip this groundwork “move fast at first but stall when gaps appear.” And the governance framing is exactly right… it’s governance for motion, not gatekeeping. Create fast paths for experimentation while your frameworks mature alongside real use cases. Once you have that, then build fluency everywhere. 2️⃣ Phase 2 is about creating organizational literacy before expecting innovation to scale. Not theoretical workshops… hands-on practice, champion networks, daily office hours to answer questions. BBVA is a perfect example: 3,000 → 11,000 ChatGPT licenses, 83% weekly usage, 2,900 GPTs created. That only happens when fluency reaches critical mass. And OpenAI calls out something most reports miss. Those SMEs you develop in Phase 2 are the ones who later define what “good” looks like as use cases get more complex. Then comes structured intake and prioritization. 3️⃣ Phase 3 tackles the “now we have 500 ideas” problem. Open intake, but disciplined scoring around impact, effort, risk, and reuse. That reuse lens is critical. Every pattern you build once should accelerate multiple future projects. Without it, you end up with duplicate agents, duplicate data, duplicate context, and duplicate risk. Finally, incremental building with continuous evaluation. 4️⃣ Phase 4 gets into the real operating rhythm… gated checkpoints (MVP → Pilot → Production), constant evaluation, and iterative improvement. OpenAI is clear here: AI systems don’t improve by scaling infrastructure. They improve by iterating the product itself… refining prompts, workflows, evals, grounding data, failure modes, cost efficiency. The core message is the same many of us have been seeing in the field. --> Organizations stuck in pilots are usually waiting for conditions that will never be perfect. --> Organizations reaching production put structure around experimentation and learn by doing. And the guide is honest about what actually drives scale: not the latest model, but the less glamorous work… building skills, capturing learnings, and creating reusable systems. Definitely worth reading if you’re in the “how do we make this repeatable?” phase rather than the “should we even be doing this?” phase.

🤖 𝐇𝐨𝐰 𝐭𝐨 𝐛𝐫𝐢𝐧𝐠 𝐀𝐈 𝐮𝐬𝐞 𝐜𝐚𝐬𝐞𝐬 𝐭𝐨 𝐥𝐢𝐟𝐞: 𝐟𝐫𝐨𝐦 𝐏𝐎𝐂 𝐭𝐨 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 I often guide businesses through implementing AI solutions. Here's a breakdown of the typical journey from concept to production. 👉 Remember: AI implementation is a step-by-step process. This enables us to learn fast and iterate. 1️⃣ 𝐏𝐫𝐨𝐨𝐟 𝐨𝐟 𝐂𝐨𝐧𝐜𝐞𝐩𝐭 (𝐏𝐨𝐂) / 𝐏𝐫𝐨𝐭𝐨𝐭𝐲𝐩𝐞 ▶ Validate the AI use case ▶ Test core functionality ▶ Gather initial feedback ❓ Key question: Is this AI solution feasible and valuable? 🖍 Example: A chatbot that can answer 5-10 basic customer queries, implemented with a few API calls and prompt engineering 2️⃣ 𝐌𝐢𝐧𝐢𝐦𝐮𝐦 𝐕𝐢𝐚𝐛𝐥𝐞 𝐏𝐫𝐨𝐝𝐮𝐜𝐭 (𝐌𝐕𝐏)/ 𝐁𝐞𝐭𝐚 𝐕𝐞𝐫𝐬𝐢𝐨𝐧 ▶ Develop core features ▶ Test with real users ▶ Gather comprehensive feedback ❓ Key question: Does this solve the problem effectively? 🖍Example: A chatbot integrated into a test website, handling 20+ common queries and basic conversations, used by a small group of beta testers. 3️⃣ 𝐆𝐨 𝐋𝐢𝐯𝐞 ▶ Scale the solution ▶ Integrate with existing systems ▶ Address security and compliance ❓ Key question: How can make the solution ready for real-world deployment? 🖍 Example: A fully integrated chatbot on the company's live website, handling thousands of queries daily, with secure data handling and 24/7 availability. 4️⃣ 𝐈𝐭𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐈𝐦𝐩𝐫𝐨𝐯𝐞𝐦𝐞𝐧𝐭 ▶ Monitor performance metrics ▶ Gather user feedback ▶ Implement improvements ❓ Key question: How can we continually enhance value? 🖍 Example: Regular updates to the chatbot, adding new features like multi-language support, integrating with CRM systems, or implementing more advanced NLP capabilities like fine-tuning of the underlying LLM. 💬 Which stage do you find most challenging? Comment below! #AIImplementation #MachineLearning #ProductDevelopment #AI --- 👩💻 I am Verena. I enable businesses to achieve AI-driven success.

Building products taught me everything about implementing AI. Most organizations approach AI backwards. They start with the technology, not the problem. You wouldn't launch a product without user research, would you? Then why deploy AI without understanding workflow pain points? The product mindset changes everything... • Start with an MVP approach • Iterate based on actual user feedback • Scale only what demonstrably works In essence, for you and me, this means we stop chasing shiny new AI tech just because it's cool. Instead, we become problem-solvers first. We ask, "What is the one thing frustrating people right now?" and then use a little bit of AI to fix just that. We then listen to see if it actually helped, and only then do we decide if it’s worth making it bigger. It's a human-centered approach to AI, ensuring it's a real tool, not a gimmick or wasted effort. AI isn't magic. It's a feature that either solves real problems or creates expensive overhead. What product principles guide your AI implementation approach? ♻️ → Repost if you found this useful! ______________ 𝗙𝗼𝗿 𝗺𝗼𝗿𝗲 𝘁𝗶𝗽𝘀, 𝗳𝗼𝗹𝗹𝗼𝘄 me: @𝗻𝗮𝘀𝘀𝗶𝗮𝘀𝗸𝗼𝘂𝗹𝗶𝗸𝗮𝗿𝗶𝘁𝗶