Virtual Environment Prototyping

This is the most important advice I gave to the class of #design students from the Delft University of Technology last week: When building an #XR prototype start by defining a clear question you want to answer: - Where will the user be once the experience starts? - How do you manage the user's attention in a #3D space? - How do you communicate certain affordances like objects that can be picked up (or not) or places that can be explored (or not)? - What will be the rough series of events the user will go through? Starting by building the "whole thing" too early will lead you to "expensive" tools like Unity, Blender or Unreal (from a time and learning curve perspective) before you have figure out and agreed on foundational aspects of your experience. This is true for students but it is also a common pitfall of more seasoned #XR teams too eager to jump into game engines without realizing the costs of rebuilding something once you realize it doesn't work well when experienced spatially. This is the main reason why I strongly believe ShapesXR is the best tool for that initial stage of design and ideation, and the fact the various groups were able to build simple environments and interactions (with sounds and haptics included 🤯) in less than 2 hours without having prior experience in Shapes is a testament to that.

This is the moment simulation becomes more important than prototyping. In our last posts, Pascalis and I showed two things: First, how you can generate a full production and warehouse environment in NVIDIA Omniverse using Claude Code and the USDA data format. Second, how NVIDIA’s new Kimodo model can generate robot motions from simple text prompts. Now we are taking the next step: Transferring robot motion into Omniverse and merging both use cases. Omniverse is not just for static visualizations. It allows dynamic simulation of movements, interactions and behavior with CAD components inside a virtual environment. And this is where it gets interesting for future product development. The vision is clear: If we can model production environments, warehouses, and real operating environments of products, we can simulate mechatronic products in realistic conditions before they physically exist. Environment → Sensor & actuator interaction → Model-in-the-loop simulation. Very similar to how autonomous vehicles are developed today, but applied to all kinds of mechatronic products. The effects are huge: • Less physical prototyping • Earlier insights without building hardware • Faster iteration cycles • Better product decisions earlier in development • Simulation becomes the main development environment Omniverse already shows how granular these simulations can be created today. Not through months of manual modeling, but increasingly through prompts that generate environments, movements and soon maybe even control logic. We are moving from designing products to designing behavior in simulated worlds first. And that will fundamentally change how we develop products. Curious to hear your thoughts! When will simulation become the primary development environment in your industry? Vlad Larichev | Rüdiger Stern | Rick Bouter | Ruben Hetfleisch | Dr.-Ing. Tobias Guggenberger

For the past few weeks, we at Dpt. have been exploring the use of generative AI workflows in a mixed reality context. The prototype I’m sharing here builds on my earlier experiments that used physical interfaces to feed and interact with a real-time img2img workflow [https://lnkd.in/eVdyaCyB]. In this iteration, I’m focusing on a first-person perspective to make the experience even more immersive. I’m not (yet 🙂) relying on a live video stream; instead, I capture a series of single snapshots from the Quest 3 passthrough feed, instantly process them with Stable Diffusion, and display the results back in the same spatial/physical location where they were taken. As Meta hasn’t yet released the Quest’s Camera API—which will give developers direct access to the device’s camera feed—I’m using the Android Media Projection API (normally used for screen recording or casting) as a temporary workaround. The diffusion workflow, exported from ComfyUI as Python, runs on a cloud GPU, letting me continue testing the prototype even when I’m outside. In the attached video, you’ll see screen recordings of me using the app at home, in the office, and outdoors. I quickly capture a series of spatial snapshots in close proximity, and once they’re processed, they form an alternate reality patchwork—the snapshots, not being perfectly aligned, create a sense of depth. You can see how my desk might look after being abandoned for years or as though it belongs in a graphic novel. You’ll also notice me spatially layering and exploring snapshots in my living room, or trying to escape the winter by recalling how Montreal’s alleyways appear in the summer, ... This meshing of virtual and physical is at the heart of what we do at Dpt. #MixedReality #AI #XR #MR #stablediffusion #rnd

Many immersive learning projects fail for the same reason many boats sink. Not because the destination was wrong. Not because the crew lacked enthusiasm. But because people jumped into the water before building the boat. In the rush to create something exciting in VR, teams often start building environments, avatars, and interactions before they have agreed on the fundamentals: -What is the learning objective? -What experience are we trying to create? -How will the learner interact with the environment? Without that foundation, development quickly turns into a cycle of endless revisions, confusion, and frustration. Over time I realized that immersive learning projects need the same thing any good construction project needs: A clear build process. That is why I use a 3-phase VR content creation protocol with clients at Craig Frehlich Consulting INC. Phase 1 — Storyboard This is where the boat is designed. We align on the learning objectives, narrative, interaction design, and educational strategy before any development begins. This is where major revisions belong. Phase 2 — MVP Walkthrough in VR Now the boat touches the water. A playable prototype allows us to test flow, pacing, and usability inside the immersive environment. At this stage we make moderate refinements to interaction and experience design. Phase 3 — Working Product Now we are sailing. The experience is polished, functional, and ready for testing. Feedback focuses on minor tweaks and bug fixes, not structural redesign. The key idea is simple: As the experience progresses, the scope of revisions narrows. This protects timelines, development resources, and the integrity of the learning design. Immersive learning can be incredibly powerful, but like any powerful tool, it works best when supported by clear instructional design processes. Otherwise, we risk doing what many teams do with new technology: Jumping into the water before the boat is built.

𝗔𝗻𝗼𝘁𝗵𝗲𝗿 𝗱𝗲𝘀𝗶𝗴𝗻 𝗿𝗲𝘃𝗶𝗲𝘄. 𝗔𝗻𝗼𝘁𝗵𝗲𝗿 𝗱𝗶𝘀𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝘄𝗵𝗮𝘁 𝘂𝘀𝗲𝗿𝘀 𝘄𝗮𝗻𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝗱𝗲𝘀𝗶𝗴𝗻𝗲𝗿𝘀 𝗯𝘂𝗶𝗹𝘁, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗰𝗮𝗻 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗺𝗮𝗸𝗲. 𝗦𝗼𝘂𝗻𝗱 𝗳𝗮𝗺𝗶𝗹𝗶𝗮𝗿? You’re not alone and it’s costing more than wasted meetings. Misalignment between users, design, and engineering is one of the biggest drivers of product delays, rework, frustration, and unexpected budget overruns. A practical way forward is emerging through Metaverse-enabled collaboration: a collaborative virtual workspace where every stakeholder can build, test, and refine ideas together in real time, with far clearer shared context. This approach is grounded in peer-reviewed research from Applied Sciences (2024) and built around a simple “Reality → Virtual → Reality” flow. 𝗥𝗲𝗮𝗹𝗶𝘁𝘆 → 𝗕𝗿𝗶𝗻𝗴 𝗲𝘃𝗲𝗿𝘆𝗼𝗻𝗲 𝗶𝗻 𝗲𝗮𝗿𝗹𝘆 Lead users, everyday users, designers, engineers, suppliers, and internal teams are identified up front. 𝗩𝗶𝗿𝘁𝘂𝗮𝗹 → 𝗠𝗲𝗲𝘁 𝗶𝗻 𝗼𝗻𝗲 𝘀𝗵𝗮𝗿𝗲𝗱 𝘀𝗽𝗮𝗰𝗲 Using VR/XR tools and digital identities, participants collaborate as digital individuals—removing geographic and timing barriers. 𝗖𝗼-𝗰𝗿𝗲𝗮𝘁𝗲 𝗮𝗰𝗿𝗼𝘀𝘀 𝗳𝗼𝘂𝗿 𝘃𝗶𝗿𝘁𝘂𝗮𝗹 𝘄𝗼𝗿𝗸𝘀𝗽𝗮𝗰𝗲𝘀 𝗜𝗱𝗲𝗮 & 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗦𝗽𝗮𝗰𝗲 Align on needs and challenges 𝗗𝗲𝘀𝗶𝗴𝗻 𝗦𝘁𝘂𝗱𝗶𝗼 Refine prototypes together 𝗩𝗶𝗿𝘁𝘂𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗭𝗼𝗻𝗲 Explore early usability insights 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗥𝗼𝗼𝗺 Blend user feedback with expert review Here’s how it works in practice: Bengaluru: A user suggests a navigation change Berlin: A designer updates the layout instantly Kuala Lumpur: A lead user explores the updated flow Detroit: Engineering checks manufacturability in minutes 𝗬𝗼𝘂𝗿 𝘁𝘂𝗿𝗻 What’s your biggest challenge in cross-functional product development? #ProductDevelopment #Innovation #DigitalTransformation #UX #Engineering

From idea → VR prototype → under 60 minutes. One of our creators walked into the office with a fun challenge: “Can I build a 3rd-person site navigation experience in an hour?” The mission: help users instantly understand a workspace before stepping into it, giving them spatial intelligence in minutes. Here’s how they did it, no code required: 📄 Typed a short brief → uploaded to VRseBuilder Studio 🤖 AI auto-generated a VR storyboard in minutes 🎮 Jumped into Unity to place & scale assets, set up a navigation path 🌐 Published & tested in VR — same day Why I am excited to share this: it’s proof that anyone, not just developers, can turn an idea into an interactive VR training as easily as editing a photo in Photoshop. From a lab to a steel plant to corporate spaces, the same template can train teams to navigate any environment. Imagine what your team could build in a day. #VR #AI #NoCode #EnterpriseTraining #SpatialIntelligence #DigitalTwin

Type a single prompt. Walk through the world you just created. In real time. That's Google's Genie 3. Not pre-rendered. Not a video. A fully interactive 3D environment generated at 24fps that remembers where you've been for minutes, not seconds. What makes this technically significant: → Visual memory up to 1 minute. Leave a location, return, it's exactly as you left it. → 720p, real-time navigation. No latency. → "Promptable world events" let you alter weather, objects, or the environment mid-session. This is a step toward unlimited training environments for robotics, simulation testing without expensive 3D modeling, and design prototyping that doesn't require building anything first. The limitation? Still capped at a few minutes of consistency. That window will expand. What catches my attention: this moves world simulation from "consume a video" to "navigate a space." Fundamentally different. Learn more here → https://lnkd.in/eacn9vX5 #AI #WorldModels #GoogleDeepMind #GenerativeAI #Robotics #Genie3 #FutureOfDesign #ProductDesign