VolumetricOR

VolumetricOR is the exploration of a concept, which allows surgical staff to learn and train surgical interventions in a photorealistic virtual operating room. Situated in the virtual reality environment, users can experience surgical workflows based on Volumetric Video recorded directly in the operating theater.

For this project, I photographed and reconstructed an operating room with Agisoft Metashape, but due to the large amount of reflective, transparent and montone surfaces in this scan, I reconstructed most of the model by hand and then re-projected the textures onto the new surfaces in Metashape.

You can use this model for free in your projects, download it on sketchfab:

The biggest challenge for this project was capturing a volumetric video of a real kidney transplantation. We had to gurantee that our setup would not obstruct the surgical stuff in any way, which meant that we had to make the setup as small as possible. Each capture unit contained a Kinect V2 depth sensor in combination with a DSLR camera, and a mini desktop PC.

In total, three of these setups were used and connected via a local Wifi network to control them. The capture was then controlled via a seperate laptop outside of the operating room.
We used DepthKit Pro and LiveScan3D as capture software, and then cleaned and imported the sequences into Unreal Engine. The quality of these captures is pretty rough, as you can see in the video at the top, or in this static frame below. This is a state from 2018 however, things improve quickly in this field.

VolumetricOR: A new Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education

Queisner M, Pogorzhelskiy M, Remde C, Pratschke J,  Sauer IM, Surgical Innovation. 2022

The complete project is availble as open-source project. Open the source files in Unreal Engine 4.19. You need a HTC-Vive, or a similar compatible SteamVR-Headset to be able to view the project in VR

Download