Adaptive Anatomy

Surgical practice usually separates between imaging and intervention: i.e. a surgeon has to apply a CT scan to the patient’s anatomy cognitively. With this concept, we want to show how this cognitive gap can be briged with the use of Augmented Reality Headsets.

A central part of our concept is the Phantom, a 3D-printed anatomic puppet, which is based on real patient data. Not only is the phantom geometry true to reality, the tissue is also true to touch, e.g. a tumor is harder than fatty tissue. This allows us to test Augmented Reality concepts on a surface which is very close to the actual conditions in the operating room. The phantom is fitted with a Marker, which allows us to detect it’s location and rotation in the real world with a Hololens 2 device.

We used the Unity3D engine in combination with the Hololens 2, to augment the liver tissue with different experimental visualisations. The Hololens 2 detects the Phantoms presence by scanning a marker positioned on the bottom of it. You can then see different visualisations that are based on the same CT-Data as the phantom puppet. The visualisations are quite experimental and represent more of a testbed, to see which challenges might appear when augmenting human tissue and how our visual system interprets them.

One interesting learning we had is, that complete X-Ray Vision, e.g. being able to always see through the phantom body onto the augmented surfaces, is heavily distracting and an unpleasant experience. Therefore we created a mask, which lets you only see the augmented content when you also directly look into the abdominal cavity, the skin being the border for the content.

Take a look a the app in action. This video was recorded using the Microsoft Mixed Reality Spectator View and is a exact mirror of what you would see inside of the Hololens 2:

We’re currently still working on the publication, please stay tuned!

The Code is freely available on our Github. You need Unity 2019.2.21f1 (exactly this version!) and a Hololens 2 device to be able to play this project:

View the Code