When Experimentation is Not the Answer

Recently I've been doing some contract work and in every engagement I get posed a similar problem, “our conversions are tanking, why”.

Most of you would guess correctly if I told you that I would suggest they need to adopt a culture of experimentation, But in the rush to test and implement, there’s a critical yet often overlooked question: is the site even ready for testing?

Sometimes, the issue isn’t about which version of an experience is better or what is the best way to address a UX friction point. Sometimes, the real problem lies buried in a website’s foundation—bugs, glitches, and errors that undermine the very principles of good user experience. Most of these go undetected as they site performs well for the business key stakeholders but has not been reviewed and tested for the range of devices, operating systems and browser versions used by customers.

Every company I know runs a form of QA before going live, however most forget to do routine perpetual QA to ensure that in the rounds of releases that everything still works like it was originally intended.

I’ve been testing now for sometime and I am always shocked at the number of bugs and defects we find. When we think of bugs, we often think of minor inconveniences—a button that doesn’t work or a page that takes a few extra seconds to load. But in reality, bugs can have massive consequences, costing businesses reputational damage and millions in missed revenue. If you think it’s only the small end of town you would be mistaken,

• Amazon experienced a pricing glitch where items across its platform were accidentally listed for as low as $1. This error wasn’t discovered until thousands of orders had already been placed, leading to significant financial losses for sellers. The glitch not only affected revenue but also eroded trust among sellers, who were left to bear the brunt of the issue.While major glitches make headlines, smaller bugs can quietly erode user experience and profitability over time.

• A health insurance provide I worked with had an Issue with their CAPTCHA check timining out meaning 50% of potential customers were not able to progress with their order. This had a both a massive revenue impact costing millions of dollars, but also a knock on effect to the call centre who had to field issues about site bugs.

• A large eCommerce supplier had a bug within their postage delivery timing causing users to abandon their cart as they could not provide shipping timeframes.

According to a Google study, 53% of mobile users abandon a site if it takes more than 3 seconds to load. Slow-loading pages, broken links, and inconsistent functionality might seem trivial but have a profound impact on key metrics like bounce rate, session duration, and conversion rates.

Experimentation works best when the environment is controlled, stable, and optimised. Running experiments on a buggy website introduces several risks:

• Skewed Results: Bugs can interfere with test outcomes, leading to misleading data. For instance, a broken form submission may show a 0% conversion rate for all variants, masking any real insights.
• Wasted Resources: Experimentation demands time, tools, and human capital. Testing in an unstable environment wastes these resources, as the insights gained are often invalid.
• Eroded Credibility: Customers encountering bugs during testing phases may lose faith in the brand, diminishing the value of any improvements made later.

It’s always tempting to dive headfirst into experimentation and fix the dropping conversion rate. But a site riddled with bugs is like a house built on a shaky foundation—no matter how beautifully you decorate it, it’s bound to crumble.

Fixing bugs, optimising performance, and ensuring a stable environment are not just technical necessities; they are business imperatives.Only when the foundation is solid can experimentation truly thrive, delivering meaningful insights and driving sustainable growth.