For a stable simulation you want your initial conditions to be stable, that's what this model optimizes for. But stability tends to be highly dependent on your settings, timestep, etc. Mostly because physics engines are designed for speed, not perfect accuracy.
I've only tackled this problem with shitty low-stakes (AI generated games) but I found heuristic post-optimization to be "good enough" -- essentially just stabilizing the simulation manually during scene bake. And predictably I had to rejigger the v-hacd and constants when I switched physics engines.
So the real usefulness of something like this is how well it generalizes to the heterogeneous simulation ecosystem, but the paper doesn't seem to cover that.
Reconstructing Physically Stable 3D Scenes from
a Single Image
Carnegie Mellon University
▼
⚡ TL;DRFrom a single casual image to a visually consistent and physically stable interactive 3D scene.
Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets for applications such as immersive interaction and content creation. However, existing single-image reconstruction methods fall short in capturing the physical structure of a scene. As a result, they often produce geometrically plausible but physically inconsistent results, including object floating and penetration, which lead to unstable behavior in physics simulations. Image-conditioned scene generation methods improve physical plausibility but often rely on strong scene priors, yielding plausible yet inaccurate object arrangements that fail to match the input image. We propose REST3D, a single-image reconstruction framework that can REconstruct physically STable 3D scenes by integrating physical scene understanding with physics-constrained refinement. We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. Leveraging this structure, we initialize the scene using image-to-3D models, followed by scene-tree-guided alignment and physics-constrained optimization to resolve physical violations while preserving visual consistency with the input image. Experiments show that our method significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets while maintaining strong reconstruction quality. We further demonstrate the reconstructed scenes in VR-based human-object interaction, showing their potential for immersive applications.
▸ Explore the physics simulation of reconstructed scenes in Isaac Gym. You can interactively rotate (drag), and zoom in (scroll) 3D to inspect the simulation process.
▸ Due to object interpenetration in baseline methods, applying gravity in the physics simulator cause objects to explosively separate and become unstable.
▸ All methods are synchronized for comparison. Scene loading may take some time.
Speed
Loading scene…
Input Image
Ours
DigitalCousins
Gen3DSR
SceneGen
SAM3D
▸ High-resolution video showing physics simulation of reconstructed scenes in Isaac Gym.
▸ Objects are placed sequentially for clarity and then simulated jointly.
▸ Our reconstructed scenes are simulation-ready which quickly settle into stable states.
We implemented an interactive VR system that reconstructs an immersive and physically grounded 3D scene from a single image, enabling users to naturally interact with stable virtual objects through hand-based interactions. Recorded with Meta Quest Pro and played back at 3× speed.
▸ Video of physics simulation of reconstructed scenes in Isaac Gym, with comparison against SOTA single-image 3D reconstruction methods.
▸ Objects are placed sequentially for clarity and then simulated jointly.
▸ Existing methods struggle to balance reconstruction fidelity and physical stability, whereas our method produces stable, simulation-ready scenes that quickly settle with only minor adjustments.
@article{ma2026rest3d,
title = {REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image},
author = {Ma, Xiaoxuan and Wang, Jiashun and Ugrinovic, Nicol\'{a}s and Litman, Yehonathan and Kitani, Kris},
booktitle = {arXiv preprint arXiv:2605.30338},
year = {2026}
}