Prototype Testing and Evaluation

One of the most common mistakes teams make when evaluating early product features is asking users whether they like an idea and treating the answer as evidence. Decades of behavioral research and very practical product research work show that this is a weak signal. People are generally bad at predicting what they will use, adopt, or pay for in the future, especially when there is no cost, effort, or tradeoff attached to their answer. That is why early feature evaluation should focus on behavior rather than belief. When a feature is only a concept, a smoke test can already tell you a lot. Exposing users to the idea through a landing page, announcement, or waitlist and observing whether they click or sign up answers a very specific question. Is this worth building at all, not whether it sounds good in theory. When an idea becomes clickable, fake door tests bring the decision closer to real behavior. Placing a realistic entry point inside the product and observing who actually tries to use it shows intent in context. The power of this method comes from the fact that users believe the feature is real at the moment of interaction. Transparency afterward is essential, but the action itself is the signal. For complex or technically risky features, especially AI, automation, or recommendation systems, Wizard of Oz prototyping allows teams to observe natural behavior before automation exists. Users interact with what looks like a fully functional system, while a human performs the work behind the scenes. This reveals expectations, decision making, and breakdowns that are invisible in abstract discussions. Concierge MVPs go one step further by making the human involvement explicit. Here, the value is delivered manually, often in a high touch way, to see whether users actually engage, return, and benefit. If people do not use or value the service when friction is low and quality is high, automation will not fix the underlying problem. Across all of these approaches, the principle is the same. Early feature evaluation should not ask people what they like. It should watch what they do when a real opportunity to engage is placed in front of them.

Two types of PMs are emerging from the AI prototyping wave. The first group learned to build. They can spin up a working prototype in 45 minutes. They demo it the next day. Stakeholders approve it because working software is more convincing than a PowerPoint. Then metrics don’t move. Nobody tested “red shoes size 10 wide” and watched the AI parse “wide” as a style descriptor. Nobody counted the clicks and realized AI search adds 2 steps over the existing filter sidebar. Nobody asked engineering about API costs at production traffic. $40K/month, unbudgeted. They went from writing bad specs to building bad prototypes. Same failure mode, just faster. The second group learned to evaluate. Boris Cherny’s Claude Code team prototyped the terminal spinner 50-100 times. 80% didn’t ship. Agent teams went through hundreds of versions. The condensed file view took 30 prototypes then a month of internal dogfooding. Boris ships 20-30 PRs a day. But the 80% he kills are more important than the 20% he ships. “Half my ideas are bad. I don’t know which half until I try.” The skill that separates these two groups is what I’m calling taste at speed: the ability to evaluate working software fast, kill most of it, and ship the survivors. A PM who reviews one spec per month builds judgment from 12 data points per year. A PM evaluating 15 prototypes per week builds judgment from 780. Same role. Same year. 65x more pattern-matching reps. That gap compounds every single week. I wrote the complete guide: 1. Why taste at speed is the defining PM skill (with the printing press analogy that changed how I think about this) 2. How Boris’s team actually works (5 parallel terminals, plan mode, phone-first agents) 3. The 5 Lenses evaluation framework (problem-solution fit, interaction cost, edge cases, technical debt, business model) 4. How to build this skill at any level (never prototyped, can prototype, ready to change your team) 5. Where the PRD fits now (it moved from step 2 to step 6) 6. A full real-world teardown showing the same feature evaluated by two PMs with wildly different outcomes Plus 4 downloadable templates: a Prototype Evaluation Scorecard, a Skill-Building Roadmap, a Prototype-First PRD Template, and a Divergent Prototyping Prompt Template. Full guide for subscribers: https://lnkd.in/g-HmamRS Not everyone can be Boris. Most PMs have meetings from 9 to 5 and a company that still requires PRDs. But a director who prototypes one feature per month makes dramatically better decisions because of it. A parent doing one prototype per sprint is already ahead of 90%. The reps compound regardless of volume.

Single prototypes tell you nothing about system reliability. Modularity is the secret key you're missing. When we built the multi-function demonstrator for Hyundai Cradle, we created a series of modular prototypes. Each targeted at validating specific performance vectors. → Thermal modules tested for uniformity and delta-T across surfaces → Touch and switch modules evaluated for actuation force versus signal-to-noise ratio → Pressure sensing modules designed to maintain accuracy under cyclic compression and lateral shear Key variables we isolated included: → Material stack-up compression profiles during environmental cycling → UV adhesive bond stability across operational temperature bands (-40°C to +85°C) → Electrical resistance drift under flexural fatigue testing (bend radius <5mm, 10,000+ cycles) By modularizing early, we could: → Identify failure modes before scaling → Fine-tune adhesives, conductors, and substrates independently → Model manufacturing tolerances with real data, not assumptions In hardware, scalable design isn’t about the first build. It’s about how you architect your prototyping process.

🔬 UX Concept Testing. How to test your UX design without spending too much time and effort polishing mock-ups and prototypes ↓ ✅ Concept testing is an early real-world check of design ideas. ✅ It happens before a new product/feature is designed and built. ✅ It helps you find an idea that will meet user and business needs. ✅ Always low-fidelity, always pre-launch, always involves real users. 🚫 Testing, not validation: ideas are not confirmed, but evaluated. ✅ What people think, do, say and feel are often very different things. ✅ You’ll need 5 users per feature or a group of features. ✅ You will discover 85% of usability problems with 5 users. ✅ You will discover 100% of UX problems with 20–40 users. 🚫 Poor surveys are a dangerous, unreliable tool to assess design. 🚫 Never ask users if they prefer one design over the other. ✅ Ask what adjectives or qualities they connect with a design. ✅ Tree testing: ask users to find content in your navigation tree. ✅ Kano model survey: get user’s sentiment about new features. ✅ First impression test: ask to rate a concept against your keywords. ✅ Preference test: ask to pick a concept that better conveys keywords. ✅ Competitive testing: like preference test, but with competitor’s design. ✅ 5-sec test: show for 5 secs, then ask questions to answer from memory. ✅ Monadic testing: segment users, test concepts in-depth per segment. ✅ Concept testing isn’t one-off, but a continuous part of the UX process. In design process, we often speak about “validation” of the new design. Yet as Kara Pernice rightfully noted, the word is confusing and introduces bias. It suggests that we know it works, and are looking for data to prove that. Instead, test, study, watch how people use it, see where the design succeeds and fails. We don’t need polished mock-ups or advanced prototypes to test UX concepts. The earlier you bring your work to actual users, the less time you’ll spend on designing and building a solution that doesn’t meet user needs and doesn’t have a market fit. And that’s where concept testing can be extremely valuable. Useful resources: Concept Testing 101, by Jenny L. https://lnkd.in/egAiKreK A Guide To Concept Testing in UX, by Maze https://lnkd.in/eawUR-AM Concept Testing In Product Design, by Victor Yocco, PhD https://lnkd.in/egs-cyap How To Test A Design Concept For Effectiveness, by Paul Boag https://lnkd.in/e7wre6E4 The Perfect UX Research Midway Method, by Gabriella Campagna Lanning https://lnkd.in/e-iA3Wkn Don’t “Validate” Designs; Test Them, by Kara Pernice https://lnkd.in/eeHhG77j UX Research Methods Cheat Sheet, by Allison Grayce Marshall https://lnkd.in/eyKW8nSu #ux #testing

Customer discovery via functional prototypes + PostHog is night & day better than the old school way of asking for feedback on Figma mockups. Here's why: I get to observe actual user behavior instead of asking the user to guess how they might use my product. My favorite example of why this matters comes from a Sony Walkman user study. They asked a bunch of people what they thought about a yellow walkman and they said "so sporty! not boring like the black one!". And yet, when they were given the opportunity to take a walkman home after the study, everyone picked the black one. We learned a lot more from user behavior than we did expressed preferences. Here's my setup for now observing user behavior from prototypes: 1. Create a functional prototype in your favorite prototyping tool (Bolt, Lovable, Reforge Build, Magic Patterns, Claude Code) 2. Ask the prototyping tool to integrate PostHog analytics 3. Ask the prototyping tool to instrument key user actions in PostHog Then you get all of these ways of observing actual behavior: - DAUs \ WAUs \ retention curves - I can actually see if people come back and use my prototype instead of taking their word for it - Action metrics dashboards - I can see what actions people are taking vs not - Post-usage survey - I can add a built-in pop-up survey to ask the user a question about the experience after they have engaged with the prototype - Session replays - I can see exactly where people are clicking and how they are using the product to identify usability issues - Heatmaps - I can see what part of my design is working across all sessions I'd never go back to testing with just a mockup after this.

User testing is a great way to get early feedback from your users. But many teams don't put much thinking into it...they jump straight to a clickable prototype (typically in Figma) and put it in front of users. Rather the teams who crush this take it up a notch! They begin with asking: - What assumptions are we testing? - Who will we test with? - How do these assumptions show up in the prototype? - What will the prototype test? What will it NOT? - How do we intend to perform the user testing? (e.g. is it virtual or in person?) From there we can begin to determine what's the best type of prototype to use. I use the below matrix with teams to help them decide and decipher the different kinds of prototypes: Depending on what you're trying to test you might want to go with high or low fidelity and also depending on your skills/access to specialist capabilities you might choose to go with a technical or low tech approach. HIGH-FIDELITY / LOW TECH e.g. Interactive Mock-Ups Testing: Usability HIGH-FIDELITY / HIGH TECH e.g. Pilot, Beta, AB/404 Tests Testing: Desirability LOW-FIDELITY / LOW TECH e.g. Wire Frames, Mock-Ups Testing: Viability LOW-FIDELITY / HIGH TECH e.g. Proof-of-Concepts Testing: Feasibility Hope that helps! Bonus, here's a template for planning your prototypes: https://lnkd.in/gV25w7WN #ProductManagement #DesignThinking #ProductDesign #ProductDiscovery

1. Foundational Research (Understanding the problem & users) User Interviews – Talking to potential or existing users to learn about their needs, challenges, and habits. Surveys & Questionnaires – Collecting quantitative and qualitative insights from a larger audience. Contextual Inquiry – Observing users in their natural environment while they use similar products or perform relevant tasks. Stakeholder Interviews – Understanding business goals, constraints, and expectations from the project. --- 2. Exploratory Research (Discovering opportunities) Market & Competitor Analysis – Studying similar products to identify gaps and best practices. Persona Creation – Summarizing different user types with their goals, frustrations, and preferences. Customer Journey Mapping – Visualizing a user’s end-to-end interaction with the product. --- 3. Generative Research (Shaping ideas) Brainstorming & Co‑Creation Workshops – Collaborating with stakeholders and users to ideate solutions. Task Analysis – Breaking down how users accomplish key tasks to find pain points. --- 4. Evaluative Research (Testing and improving designs) Usability Testing – Asking users to perform tasks on a prototype or live product to spot friction points. A/B Testing – Comparing two design variations to see which performs better. Card Sorting – Understanding how users group information to improve navigation and information architecture. Tree Testing – Testing the hierarchy and structure of menus before final design. Eye-Tracking Studies – Observing where users visually focus to optimize layout and hierarchy. --- 5. Continuous Feedback & Analytics (Post-launch improvement) Heatmaps & Click Tracking – Seeing where users interact most. Analytics Review – Studying user behavior data (bounce rates, session times, funnels). Feedback Forms & Support Tickets – Gathering ongoing user feedback to refine the product.

Product managers used to overbuild in pursuit of perfection. Then we overcorrected, with raw MVPs. Today, AI prototyping gives us the tools to build better products—faster, and with more confidence. For years, validating ideas early was the goal—but it took too long. So we skipped discovery. We overbuilt based on gut. And we launched late—only to learn we were wrong. Then came MVPs. We shipped faster—but often learned less. Too lean to deliver value. Too early to earn trust. Today, there’s a better way: AI prototyping is unlocking the Build Smarter Loop. It’s a faster, more confident path to product learning: 1️⃣ Prototype to test assumptions -> Use AI prototyping tools (like v0, Bolt, Replit, Lovable) to quickly mock up key flows, feature ideas, and messaging. -> Validate your riskiest assumptions with internal teams, user testing platforms, or lightweight customer interviews—before you involve engineers. 💡 Catch bad bets early and explore multiple options without heavy lift. 2️⃣ Deliver a better product—faster and with more confidence -> Ship a lean version designed to validate learning goals, not just to “check the MVP box.” -> Because your discovery was fast and informed, your build is focused, intentional, and aligned. 💡 You launch faster without guessing—and with buy-in from users and stakeholders. 3️⃣ Learn and refine continuously -> Instrument usage to track how users interact with your product—ignore what they say, watch what they do. -> Close the loop by feeding these insights back into both your roadmap and your next round of prototyping. 💡 Every iteration gets sharper, driven by data—not gut feel. Final thought: AI prototyping enables you to improve what you launch—and how quickly you learn from it. — 👋 I’m Ron Yang, a product leader and advisor. Follow me for insights on product leadership & strategy.

What's the right number of prototype iterations in MedTech? Hint: the answer is not 1. That's like expecting a hole-in-one on a long distance golf shot. Just ain't gonna happen. Instead, focus your prototype iterations on answering specific questions: ➡️ Prototype 1: Does it work on the bench? Simplified proof-of-concept prototype that addresses key questions related to technical performance. ➡️Prototype 2: Does it work (pre)clinically? Early prototype aimed at collecting data, preclinically (for significant risk devices) or clinically (for non-significant risk devices). ➡️Prototype 3: Will people use it correctly? Usability prototypes (or mockups) aimed at evaluating user interfaces, usability, and possible misuse through human factors studies. ➡️Prototype 4: Does it achieve target COGS? Alpha prototype integrating industrial design and engineering, while designing for production materials and processes. ➡️ Prototype 5: Does it meet the requirements? Beta prototype addressing shortcomings of Alpha, and used for engineering verification testing (before V&V). So, minimum.. 5 prototype iterations. Often many more. Stage these prototype iterations so that each one gains the benefit of the prior. If you isolate these risk factors, your prototypes can be much simpler, faster and more cost effective to design, produce, and test. Prototyping is a mindset -- it's about learning, quickly and effectively. > Identify the right questions to answer > Build simple prototypes focused on the key questions > Run the tests, learn, and iterate. #medtech #medicaldevices #prototyping