Robo-Portation! A mixed reality system for creating seamless virtual-physical object manipulation illusions

Background

Mixed reality (MR) is an emerging field that blends virtual and physical environments, enabling users to interact with digital content as if it were part of their real-world surroundings. This technology holds immense promise for applications ranging from entertainment and education to remote collaboration and adaptive environments. However, the seamless integration of virtual and physical elements remains a significant challenge. Users expect virtual objects to behave with the immediacy and flexibility of digital media, yet also desire tangible, physical feedback and real-world consequences. Bridging this gap is crucial for creating truly immersive and believable MR experiences, where the boundaries between what is virtual and what is real become indistinguishable.

Current approaches to synchronizing virtual and physical object manipulations in MR environments face several limitations. Most systems rely on visible robotic actuators or manual intervention to move physical objects, which breaks the illusion of magical or instantaneous virtual actions. Additionally, existing methods for hiding or masking robots in the user’s view are often computationally intensive, visually imperfect, or require highly controlled environments. These constraints limit the types of interactions and illusions that can be achieved, reducing the sense of presence and immersion. Furthermore, precise spatial alignment and real-time synchronization between virtual animations and physical actions are technically demanding, often resulting in noticeable lags or mismatches that disrupt the user experience. As a result, the full potential of mixed reality for creating seamless, magical object manipulations remains largely unrealized.

Technology

Reality Promises is an advanced mixed reality (MR) system that enables users to experience magical, seamless manipulations of physical objects in their environment through the illusion of virtual-physical decoupling. The system combines a video-passthrough MR headset for the user, a mobile robot equipped with a second headset for precise tracking, and a sophisticated communication infrastructure. The robot, guided by a ROS2/Nav2 pipeline and analytical inverse kinematics, covertly moves physical objects while remaining invisible to the user thanks to a novel robot-aware 3D Gaussian Splatting rendering technique. This technique dynamically captures and renders only the parts of the physical scene that cover the robot, using multi-cone casting and custom shaders to blend the visuals seamlessly into the passthrough view. Users interact with virtual clones of physical objects, performing gestures to summon, relocate, or materialize items, while the system synchronizes these virtual animations with the robot’s hidden actions to create the illusion of instantaneous, fantastical object manipulations.

What differentiates this technology is its comprehensive integration of robotics, computer vision, and real-time graphics to achieve true virtual-physical decoupling illusions. Unlike traditional MR systems that merely overlay virtual content, Reality Promises physically fulfills virtual manipulations by synchronizing a hidden robot’s actions with the user’s virtual experience, maintaining a continuous sense of physical presence. The system’s robot-aware Gaussian Splatting ensures the robot remains visually concealed, allowing for effects like telekinesis, dematerialization, and magical character-driven object movement. Its flexible interaction protocols, surface-aware gesture controls, and robust safety measures enable a wide range of applications, from immersive entertainment and remote collaboration to health and responsive architecture. By tightly coupling physical robotics with virtual perception and rendering, Reality Promises pushes the boundaries of MR, realizing illusions where computers can control the apparent existence and behavior of matter in the user’s environment.

Benefits

• Enables seamless illusions of magical object manipulations by synchronizing invisible robot actions with virtual reality
• Creates virtual-physical decoupling allowing users to perceive instantaneous, fantastical object movements physically fulfilled by robots
• Uses advanced robot-aware 3D Gaussian Splatting to invisibly render robots in mixed reality passthrough views
• Supports natural, intuitive interaction with distant and local objects via hand gestures and virtual overlays
• Facilitates immersive mixed reality experiences such as telekinesis, materialization, and virtual character-driven object manipulation
• Integrates precise robot navigation and manipulation with real-time pose tracking for accurate physical fulfillment
• Applicable across entertainment, remote collaboration, health nudging, and responsive architectural environments
• Enhances user immersion by maintaining consistent spatial alignment and seamless recoupling of virtual and physical objects

Applications

• Immersive MR gaming experiences
• Remote physical workspace manipulation
• Interactive MR storytelling installations
• Behavioral nudging in smart homes
• Adaptive physical retail displays

Additional info

This mixed reality system creates illusions of magical physical object manipulation. It synchronizes hidden mobile robot actions, physically moving objects, with virtual perceptions via video-passthrough headsets. A robot-aware 3D Gaussian Splatting technique renders only splats covering the robot, making it invisible, while shared spatial anchors enable precise tracking and control for seamless virtual-physical decoupling.