Balancing Experimentation and Execution

The usual thinking often goes, "We're changing the website/platform, so there's no point optimizing what we already have." This perspective, while common, can inadvertently equate experimentation solely with optimisation, potentially overlooking the enormous benefits of integrating a truly experimental approach into development and innovation. A replatforming or redesign project typically involves a complex decision-making and MoSCoW-style exercise centered around a set of features. It's often impossible to exactly replicate old features on a new platform, meaning crucial decisions must be made about what's essential and what might be dropped. Likewise, new platforms can introduce various potential new features, but are they truly worth the investment? These decisions can become complex, political, and increasingly stressful as deadlines loom. The risk is that choices are made based on internal influence rather than what will genuinely serve the customer, which is inherently difficult to guess. How can you better manage this process? How can you genuinely know what will deliver the best customer experience and commercial outcomes? EXPERIMENTATION! When done properly, experimentation (including but not limited to A/B testing) can fast-track this entire process and help you deliver a project that actually works. Consider starting by creating a comprehensive list of all feature disparities that need to be addressed. Then, establish an initial prioritization. Next, plan and run experiments for each consideration. Finally, assess the likely benefit. Some experiments are remarkably straightforward. If a new platform won't include a particular feature "out of the box," you could A/B test removing it from your existing site to understand its true importance. Others might be more challenging. If a new platform offers recommendations but at additional cost, you could conduct more rudimentary experiments on your existing site to test the core concept. Moreover, these features don't have to be front-end; the same process can be applied to backend operational features if you have the right expertise. Experimentation isn't just optimisation; it's a critical tool for informed innovation. #experimentation #cro #productmanagement #growth #digitalexperience #experimentationledgrowth #elg #growthexperimentation

A $100B company was built on blurry photos. Until three founders flew to New York with cameras. In the early days of Airbnb, listings had terrible photos. Blurry rooms. Dark apartments. Low trust. The founders formed a hypothesis: Professional photos will increase bookings. They did not commission a strategy deck. They grabbed cameras. Tested it manually. Result: Listings with professional photos received more than 2x bookings. One experiment. A new growth engine. That mindset scaled. Airbnb increased testing velocity from fewer than 100 experiments per week to more than 700 per week. But the real breakthrough was alignment. They chose one north star metric: Nights Booked Not clicks. Not impressions. Not feature adoption. Every experiment answered one question: Did it increase nights booked? Examples: ✔️ Simple UI test Opening listings in a new tab → 3 to 4% increase in bookings ✔️ A/A tests Control vs control → Ensured the experimentation system itself was trustworthy ✔️ Internal Experiment Reporting Framework Automated analysis → Limited bias → Standardized review Lessons for operators: ➕ Start with a hypothesis, not a feature ➕ Hire data talent early ➕ Embed analysts in product teams ➕ Align everyone to one metric ➕ Validate your testing system Data is not a dashboard. It is the customer’s voice. I share insights on how experimentation fuels innovation and transforms company culture. Follow me for more on building a culture of curiosity.

“We’ve always done it that way” is the first line of a recipe for never moving forward. “Still works” is seductive. Familiar processes feel efficient, and habits that once produced results look rational. But the operating question has changed from “Does it work?” to “Is it still the best way?” In high-change environments, yesterday’s wins become today’s ceilings. Research backs this up. McKinsey describes the “adaptability paradox”: precisely when leaders need fresh thinking, they default to the familiar, which blocks the learning and innovation required for what comes next. Adaptability is what moves organizations from enduring shocks to thriving beyond them. McKinsey also notes that resilience and adaptability are distinct. Resilience responds well to events; adaptability rewires how you operate so you can keep creating advantage under new conditions. That means systems, skills, and metrics must evolve in sync with the environment, not after it. Harvard Business Review puts a sharper point on it: the obstacle isn’t learning new things, it’s unlearning outdated mental models. Letting go is a skill. The cycle is: recognize what’s obsolete, install a better model, then ingrain the new habits. For operators and founders, the discipline is deciding what to stop optimizing, which dashboards to retire, and when a “steady” process is actually slowing decision speed. Stability and stagnation can look identical from the outside. The best leaders make reassessment a habit. They pressure-test assumptions on a rhythm, measure learning velocity alongside output, and design for reversibility so experimentation is safe. Unlearning is not anti-execution. It is execution for a faster game. When you subtract the now-counterproductive, you release resources for what compounds next. That’s how you convert disruption into durable advantage. Start here: ✅ This week, pick one “works fine” process and ask: is it still our best path or just our most comfortable one? ✅ Replace one “best practice” with a “next practice” built for the conditions you actually face now. ✅ Add one metric that rewards iteration and reversibility (e.g., experiments shipped, time to decision reversal). ♻️Repost & follow John Brewton for content that helps. ✅ Do. Fail. Learn. Grow. Win. ✅ Repeat. Forever. ⸻ 📬Subscribe to Operating by John Brewton for deep dives on the history and future of operating companies (🔗in profile).

Most teams are drowning in optimization test ideas... but starving for real impact. I've seen this pattern destroy more optimization programs than poor execution ever could. The problem isn't lack of creativity. It's lack of strategy. Before you run another A/B test, ask yourself four critical questions: ↳ Is this strategically important to your business goals? ↳ Are you confident the change won't harm the user experience? ↳ Can you reach statistical significance in a reasonable timeframe? ↳ Do you have the technical capability to execute properly? If any answer is "no," you have better options: ↳ De-prioritize non-strategic tests. Add them to your backlog for later consideration. ↳ Run rapid sentiment tests or task completion analysis for quick validation. Only commit to full experimentation when all four criteria align. Or implement proven solutions directly when you're confident in the outcome. This decision framework has helped our clients at The Good generate over $100 million in additional revenue by focusing their testing efforts where they matter most. Your optimization program isn't measured by how many tests you run. It's measured by how much value you create.

"Slow Down to Speed Up" – Balancing Velocity and Quality in Experimentation I’m working on a program with some QA/QC issues right now. We’ve been pushing velocity hard—really focusing on getting experiments out fast. But now it’s time to step back and reset. This happens, and it should be expected. When you’re scaling an experimentation program, it’s easy to fall into the velocity trap: rushing to ship more tests without maintaining the foundation that makes them meaningful and reliable. But here’s the thing—scaling doesn’t mean choosing between velocity and quality. It means building a system that lets you do both. Here’s how I’m stepping back to speed up: >> Refining Processes Without Creating Bottlenecks: Processes should enable speed, not slow it down. We're revisiting our workflows to ensure they support velocity while maintaining rigor—standardized, yet flexible. >> Prioritizing High-Impact Testing: Not all experiments are worth the rush. By tying experiments to KPIs and business goals, we’re focusing our resources on what truly matters, not just what’s easy to test. >> Fixing Gaps in Skills and Knowledge: When teams are pushed too hard, QA issues pop up. Auditing capabilities and addressing weak spots—whether through training, hiring, or collaboration—is key to avoiding slowdowns later. >> Rethinking Rituals to Build Collaboration: Velocity often isolates teams into silos which breeds mis-alignment. Regular reviews, cross-team standups, and shared insights can keep the momentum strong while ensuring consistency across the program. >> Optimizing Tools with XOS: An Experimentation Operating System (XOS) is helping us integrate tools, automate repetitive tasks, and give teams the resources they need to move fast without cutting corners. #systemsthinking Sometimes, scaling means pushing hard. Other times, it’s about resetting the foundation so you can move even faster later. Experimentation is a learning process—for the teams running it just as much as for the business. If you’ve ever hit QA bumps while scaling velocity, know you’re not alone. This is part of the process. What’s your approach when you need to “slow down to speed up”? Let’s share ideas below! 👇

World-class experimenters don’t celebrate wins the way most do. Here’s what they do differently 👇 They leverage one of the most impactful mental models I’ve seen for prioritising experiments: *Explore vs Exploit* In this context: 🔍 Explore → Testing in new areas to discover fresh opportunities. 🚀 Exploit → Doubling down where you’ve already found wins to maximise impact. Here’s how it works in practice: 1️⃣ You start in explore mode, running experiments across new ideas and domains. 2️⃣ You find traction. Something clearly works. 3️⃣ You switch to exploit mode, going deep with follow-on experiments to capture the full upside. 4️⃣ As gains plateau, you increase explore again to find the next big thing. It’s a wave pattern over time: explore, exploit, explore, exploit. And the balance constantly shifts as you learn. In my experience, world-class experimenters use this framework to: ✅ Deliver larger gains over time. ✅ Maintain momentum instead of chasing endless “new ideas.” ✅ Build a clearer, compounding understanding of what really moves the needle. Teams that don’t? They bounce from one idea to the next, never fully capturing the value of their wins. How do you balance explore vs exploit in your experimentation program? 👇 I’d love to hear your approach. - - - P.S. I unpack frameworks like this in The Experimenter’s Advantage, my newsletter on building high-impact experimentation programs.

Innovation dies where execution starts. Or does it? Balancing innovation and execution is a hard challenge for CEOs. Focus too much on one, and the other suffers. The best organizations find synergy between the two. And thrive because of it. Here’s how to foster a culture of innovation without sacrificing execution: 1️⃣ Define Clear Strategic Boundaries ↳ Innovation needs direction to thrive. ↳ Without focus, resources are wasted on misaligned ideas. 💡Set clear priorities so creative efforts moves your business forward. 2️⃣ Build Dedicated Spaces for Experimentation ↳ Creativity often struggles under the weight of rigid processes. ↳ Innovation needs its own room to breathe. 💡Create programs to test new ideas outside of daily operations. 3️⃣ Empower Smart Risk-Taking ↳ Fear of failure discourages creativity. ↳ Calculated risks spark breakthroughs and move organizations forward. 💡 Reward efforts that show bold thinking. 4️⃣ Recognize and Celebrate Innovation ↳ Innovation thrives on acknowledgment. ↳ Employees need to see that creativity is valued, not just execution. 💡Publicly recognize teams or individuals who think outside the box and bring fresh solutions. 5️⃣ Ensure Accountability in Execution ↳ Ideas without follow-through are wasted potential. ↳ Creativity must translate into action. 💡Assign clear ownership for delivering results from new initiatives. 6️⃣ Bridge Functions with Collaboration ↳ Without teamwork, innovation stays siloed. ↳ Collaboration ensures ideas flow seamlessly to execution. 💡 Connect innovation teams with leaders to strengthen execution. 7️⃣ Protect Time for Both ↳ Execution tends to dominate schedules. ↳ Innovation must have intentional space to grow alongside delivery. 💡 Schedule dedicated time for creative thinking and innovation. Great companies know how to balance creativity with results. It’s your job to build systems that support both. While empowering your team to turn bold ideas into real outcomes. __ How do you strike the balance between innovation and execution? ♻️ Helpful? Please repost to your network

For a decade, founders were told: “Do one thing. Stay focused. Don’t spread yourself thin.”   That made sense when building was slow and expensive. But on the road to superintelligence, I see that advice quietly stopped working.   Every day, the economics of value creation are changing: → The cost of launching a product is approaching zero. → The cost of not experimenting is rising fast.   I saw this playbook years ago in gaming — and I’m betting we’re entering a new era where every industry starts to look like a studio.   Studios like Zynga, Voodoo, and Supercell mastered this approach: Build in parallel → launch fast → track signal → kill what doesn’t stick → double down on what works → repeat.   And if you look closely, this mindset is already spreading: → Bending Spoons scales consumer apps through shared infrastructure and rapid testing. → ByteDance runs hundreds of micro-products, with algorithms deciding what lives or dies. → Shopify treats merchant tools like a sandbox — build, ship, learn, evolve. They don’t call themselves studios.   But the road to superintelligence made this strategy inevitable. Because AI collapses time-to-build, time-to-test, and time-to-learn: → Generative engineering → Instant UX + prototyping → Automated GTM + ops → Real-time signal detection   When creativity is infinite and execution is automated, feature optimization isn’t enough. You have to invent daily. Experiment, learn, double down, repeat.   At Kalder | Turn Rewards into Revenue we’ve lived a trailer of this shift firsthand. Last year, we built 18+ loyalty modules, shipped quickly, and let the data guide us. One stood out — Partnership Rewards.   A year later, we were partnering with top airlines, grocers, and fintechs to sell their rewards to 35,000+ brand partners.   And some experiments turned into unexpected hits — like viral “mystery box” subscriptions that drove $200K+ ARR per e-commerce brand we worked with, with 50%+ margin recovery.   We’re not a studio — but the motion is clear.   The old playbook was: focus on one product until it works. The new one: run a portfolio of bets until something truly breaks out — then scale it with conviction. This is how I bet the next generation of category-defining companies will be built.

When strategy and execution compete for your attention, both suffer. I learned this the hard way. At first, we approached our NSF strategic plan the same way we tackled our daily execution - together, as a full team. We assumed that having more people involved would lead to a stronger outcome. Instead, it led to bottlenecks, distractions, and inefficiency. The team was stretched thin, trying to execute programs while also shaping long-term strategy. We weren’t making enough progress on either front. Our approach simply wasn’t working. That’s when Alan Rudolph helped me see the problem from a different angle. The strategic planning process itself was becoming disruptive to productivity. It needed its own dedicated team focused solely on strategy and deliverables, while everyone else kept operations moving forward. It was a defining leadership moment for me. I realized my job isn’t to somehow make everyone do everything - it’s to trust my team, point them in the right direction, and let them execute. Now, we have a dedicated team leading strategic planning and an operations team free to do what they do best, and the difference is already clear. We’re moving forward with focus, speed, and clarity that we couldn’t achieve when we were trying to keep both plates spinning. For lean organizations balancing execution and strategy, remember: Not everyone needs to be in the room for every decision. Assign the right people, trust them, and just let your operators operate.