Quickstart | Documentation | Unity Asset Store | Unity Forums Thread | License

The Geometry Sequence Player is a package for the Unity game engine which enables playback of large geometry sequences inside of Unity. The package can be used to playback pointcloud, mesh or textured mesh sequences.

It is available on either Github (for non-commercial use) or the Unity Asset Store (for commercial use):

What is a geometry sequence?

In a geometry sequence, each frame consists out of an individual mesh or pointcloud, which is shown at short Intervalls to create the illusion of an animation, kind of like a three dimensional flipbook. As each frame is independent from the next one, no limitations apply to your animation, unlike for example skinned meshes or blendshapes, where mesh topology can’t be changed. This applies for example to:

Pre-Rendered fluid caches
Baked Physiscs simulations
Animated Procedural meshes
4D scans with per frame textures
Volumetric videos
Pointclouds captured by RGBD sensors (Kinect, Realsense, Lidar)
Data visualisations

Features

• Can import sequences from almost any source, with an included converter that covers most standard file formats.
• Easy setup of sequences with an easy-to-use Editor UI and playback controls
• Integrated into the Unity Timeline
• Granular playback control for interactive experiences is provided through a scripting API
• Sequences don’t need to be loaded into your Unity project or even the RAM, everything is streamed at runtime from disk
• Highly efficient playback system that allows you to stream millions of points/polygons per second, even on lower end hardware

The package

Originally, this package was created to facilitate the playback of volumetric videos from LiveScan3D, captured with Depth sensors such as the Azure Kinect or Intel realsense. But I realized that it could also be useful for many more applications, such as getting fluid and physics simulations from Blender into Unity!

I have used the package intensively in many projects myself, so it’s well proven and tested. The package has been released on Github as open-source code for quite a while already, but to make the development and support a bit more sustainable, I now decided to also offer the package on the Unity Asset Store.
If you want to use the package commercially, or if you want to support our development, you can buy it there! Thank you very much for your support

Read Article on Frontiers (Open Acces)
Download PDF

Abstract

Introduction

Volumetric video production in commercial studios is predominantly produced using a multi-view stereo process that relies on a high two-digit number of cameras to capture a scene. Due to the hardware requirements and associated processing costs, this workflow is resource-intensive and expensive, making it unattainable for creators and researchers with smaller budgets. Low-cost volumetric video systems using RGBD cameras offer an affordable alternative. As these small, mobile systems are a relatively new technology, the available software applications vary in terms of workflow and image quality. In this paper we provide an overview of the technical capabilities of sparse camera volumetric video capture applications and assess their visual fidelity and workflow.

Materials and methods

We selected volumetric video applications that are publicly available, support capture with multiple Microsoft Azure Kinect cameras and run on consumer-grade computer hardware. We compared the features, usability, and workflow of each application and benchmarked them in five different scenarios. Based on the benchmark footage, we analyzed spatial calibration accuracy, artifact occurrence and conducted a subjective perception study with 19 participants from a game design study program to assess the visual fidelity of the captures.

Results

We evaluated three applications, Depthkit Studio, LiveScan3D and VolumetricCapture. We found Depthkit Studio to provide the best experience for novel users, while LiveScan3D and VolumetricCapture require advanced technical knowledge to be operated. The footage captured by Depthkit Studio showed the least amount of artifacts by a larger margin, followed by LiveScan3D and VolumetricCapture. These findings were confirmed by the participants who preferred Depthkit Studio over LiveScan3D and VolumetricCapture.

Discussion

Based on the results, we recommend Depthkit Studio for the highest fidelity captures. LiveScan3D produces footage of only acceptable fidelity but is the only candidate that is available as open-source software. We therefore recommend it as a platform for research and experimentation. Due to the lower fidelity and high setup complexity, we recommend VolumetricCapture only for specific use-cases where its ability to handle a high number of sensors in a large capture volume is required.

Read article on IEEE
Download PDF

Abstract

This study presents a mixed reality training concept designed to enhance medical students’ acquisition of surgical knot-tying skills, a fundamental component of surgical training critical for effective wound closure and tissue healing. Utilizing a virtual reality headset with video passthrough functionality, the system provides adaptive visual instructions tailored to the user’s hand movements during the knot-tying process. A prototype was developed based on the concept, featuring three-dimensional videos in which virtual instructor hands demonstrate each step of the procedure. The training concept was derived from an iterative, user-centered process encompassing requirement analysis, prototype development, and evaluation. Key functionalities include the ability to display thread tension and tensile strength, dynamically adapt learning speed to the user’s progress, and deliver personalized feedback by visually augmenting the hands and fingers. Evaluation results indicate that spatial and tangible interactions facilitated by the mixed reality training prototype support the acquisition of practical skills, bridging the gap between digital and physical simulation training.

Read Article on ResearchGate
Download PDF

Abstract

Background:

Tying surgical knots is a basic but critical skill that surgeons must master. Currently, knot tying is typically taught through instructor observation or instructional videos. These methods are either very resource consuming or offer little interactivity and customizability. Knot tying training in extended reality (XR) could address these weaknesses and improve the linking of observation and application: For example, spatial awareness and the ability to mentally rotate, i.e. imagine knots from different perspectives, affects learning performance (Brandt & Davies, 2006). XR may support spatial awareness by allowing knot tying from different perspectives superimposed on the real world.

Method:

The project GreifbAR develops an XR-based interactive knot tying training application. This application teaches the process of knot tying and provides individualized feedback based on hand pose and scene recognition. To achieve a learning-friendly design and user acceptance, the requirements of learners and experts must be considered. Therefore, based on a literature review, an online survey with 80 medical students and four interviews with experienced surgeons at Charité – Universitätsmedizin Berlin were conducted.

Initial results:

The respondents show openness towards knot tying training with XR, yet emphasize the importance of a realistic learning situations and personal guidance. They also report little prior experience with XR. The talk integrates the survey results with findings from technology acceptance research and derives implications for the design of XR-based training systems for knot tying and similar procedural tasks.

Read Article on Surgical Innovation
Download PDF

Abstract

Background

Surgical training is primarily carried out through observation during assistance or on-site classes, by watching videos as well as by different formats of simulation. The simulation of physical presence in the operating theatre in virtual reality might complement these necessary experiences. A prerequisite is a new education concept for virtual classes that communicates the unique workflows and decision-making paths of surgical health professions (i.e. surgeons, anesthesiologists and surgical assistants) in an authentic and immersive way. For this project, media scientists, designers and surgeons worked together to develop the foundations for new ways of conveying knowledge using virtual reality in surgery.

Materials and method

A technical workflow to record and present volumetric videos of surgical interventions in a photorealistic virtual operating room was developed. Situated in the virtual reality demonstrator called VolumetricOR, users can experience and navigate through surgical workflows as if they are physically present. The concept is compared with traditional video-based formats of digital simulation in surgical training.

Results

VolumetricOR let trainees experience surgical action and workflows (a) three-dimensionally, (b) from any perspective and (c) in real scale. This improves the linking of theoretical expertise and practical application of knowledge and shifts the learning experience from observation to participation.

Discussion

Volumetric training environments allow trainees to acquire procedural knowledge before going to the operating room and could improve the efficiency and quality of the learning and training process for professional staff by communicating techniques and workflows when the possibilities of training on-site are limited.

Read Article on Wiley
Download PDF

Abstract

Recent developments in the field of augmented reality (AR) have enabled new use cases in surgery. Initial set-up of an appropriate infrastructure for maintaining an AR surgical workflow requires investment in appropriate hardware. We compared the usability of the Microsoft HoloLens and Meta 2 head mounted displays (HMDs). Fifteen medicine students tested each device and were questioned with a variant of the System Usability Scale (SUS). Two surgeons independently tested the devices in an intraoperative setting. In our adapted SUS, ergonomics, ease of use, and visual clarity of the display did not differ significantly between HMD groups. The field of view (FOV) was smaller in the Microsoft HoloLens than the Meta 2 and significantly more study subjects felt limited through the FOV. Intraoperatively, decreased mobility due to the necessity of an AC adapter and additional computing device for the Meta 2 proved to be limiting. Object stability was rated superior in the Microsoft HoloLens than the Meta 2 by our surgeons and lead to increased use. In summary, after examination of the Meta 2 and the Microsoft HoloLens, we found key advantages in the Microsoft HoloLens which provided palpable benefits in a surgical setting.

Back to Publication List