Quickstart | Documentation | Unity Asset Store | Unity Forums Thread | License

The Geometry Sequence Player is a package for the Unity game engine which enables playback of large geometry sequences inside of Unity. The package can be used to playback pointcloud, mesh or textured mesh sequences.

It is available on either Github (for non-commercial use) or the Unity Asset Store (for commercial use):

What is a geometry sequence?

In a geometry sequence, each frame consists out of an individual mesh or pointcloud, which is shown at short Intervalls to create the illusion of an animation, kind of like a three dimensional flipbook. As each frame is independent from the next one, no limitations apply to your animation, unlike for example skinned meshes or blendshapes, where mesh topology can’t be changed. This applies for example to:

Pre-Rendered fluid caches
Baked Physiscs simulations
Animated Procedural meshes
4D scans with per frame textures
Volumetric videos
Pointclouds captured by RGBD sensors (Kinect, Realsense, Lidar)
Data visualisations

Features

• Can import sequences from almost any source, with an included converter that covers most standard file formats.
• Easy setup of sequences with an easy-to-use Editor UI and playback controls
• Integrated into the Unity Timeline
• Granular playback control for interactive experiences is provided through a scripting API
• Sequences don’t need to be loaded into your Unity project or even the RAM, everything is streamed at runtime from disk
• Highly efficient playback system that allows you to stream millions of points/polygons per second, even on lower end hardware

The package

Originally, this package was created to facilitate the playback of volumetric videos from LiveScan3D, captured with Depth sensors such as the Azure Kinect or Intel realsense. But I realized that it could also be useful for many more applications, such as getting fluid and physics simulations from Blender into Unity!

I have used the package intensively in many projects myself, so it’s well proven and tested. The package has been released on Github as open-source code for quite a while already, but to make the development and support a bit more sustainable, I now decided to also offer the package on the Unity Asset Store.
If you want to use the package commercially, or if you want to support our development, you can buy it there! Thank you very much for your support

Read Article on Frontiers (Open Acces)
Download PDF

Abstract

Introduction

Volumetric video production in commercial studios is predominantly produced using a multi-view stereo process that relies on a high two-digit number of cameras to capture a scene. Due to the hardware requirements and associated processing costs, this workflow is resource-intensive and expensive, making it unattainable for creators and researchers with smaller budgets. Low-cost volumetric video systems using RGBD cameras offer an affordable alternative. As these small, mobile systems are a relatively new technology, the available software applications vary in terms of workflow and image quality. In this paper we provide an overview of the technical capabilities of sparse camera volumetric video capture applications and assess their visual fidelity and workflow.

Materials and methods

We selected volumetric video applications that are publicly available, support capture with multiple Microsoft Azure Kinect cameras and run on consumer-grade computer hardware. We compared the features, usability, and workflow of each application and benchmarked them in five different scenarios. Based on the benchmark footage, we analyzed spatial calibration accuracy, artifact occurrence and conducted a subjective perception study with 19 participants from a game design study program to assess the visual fidelity of the captures.

Results

We evaluated three applications, Depthkit Studio, LiveScan3D and VolumetricCapture. We found Depthkit Studio to provide the best experience for novel users, while LiveScan3D and VolumetricCapture require advanced technical knowledge to be operated. The footage captured by Depthkit Studio showed the least amount of artifacts by a larger margin, followed by LiveScan3D and VolumetricCapture. These findings were confirmed by the participants who preferred Depthkit Studio over LiveScan3D and VolumetricCapture.

Discussion

Based on the results, we recommend Depthkit Studio for the highest fidelity captures. LiveScan3D produces footage of only acceptable fidelity but is the only candidate that is available as open-source software. We therefore recommend it as a platform for research and experimentation. Due to the lower fidelity and high setup complexity, we recommend VolumetricCapture only for specific use-cases where its ability to handle a high number of sensors in a large capture volume is required.

Read article on IEEE
Download PDF

Abstract

This study presents a mixed reality training concept designed to enhance medical students’ acquisition of surgical knot-tying skills, a fundamental component of surgical training critical for effective wound closure and tissue healing. Utilizing a virtual reality headset with video passthrough functionality, the system provides adaptive visual instructions tailored to the user’s hand movements during the knot-tying process. A prototype was developed based on the concept, featuring three-dimensional videos in which virtual instructor hands demonstrate each step of the procedure. The training concept was derived from an iterative, user-centered process encompassing requirement analysis, prototype development, and evaluation. Key functionalities include the ability to display thread tension and tensile strength, dynamically adapt learning speed to the user’s progress, and deliver personalized feedback by visually augmenting the hands and fingers. Evaluation results indicate that spatial and tangible interactions facilitated by the mixed reality training prototype support the acquisition of practical skills, bridging the gap between digital and physical simulation training.

Hallo Media Lab Bayern!

Vielen Dank dass ihr euch meine Bewerbung anschaut! Wenn ihr daran interessiert seit einen aktuellen Prototypen dieses Projekts einmal genauer unter die Lupe zu nehmen, seid ihr hier richtig! Zum testen des Prototypen benötigt ihr einen Windows-PC.

Download LiveScan3D Prototype

LiveScan3D nimmt das volumetrische Video mithilfe spezieller Kameras, sogennanter Tiefenkameras, oder RGBD-Kameras auf. Diese Kameras nehmen zusätzlich zu der Farbe (RGB) auch die Distanz (D) eines jeden Pixels zur Kamera auf. Momentan wird von LiveScan3D nur das Model Microsoft Azure Kinect unterstützt.

Ihr habt zufällig eine Azure Kinect bei euch? Großartig, dann müsst ihr nur noch das Azure Kinect SDK installieren und die Kinect per USB mit eurem PC verbinden.

Ihr habt keine Kinect? Kein Problem! Ich habe euch eine spezielle Version des Prototypen erstellt, in dem die Kinects einfach simuliert werden, ihr benötigt also nichts weiter.

Anleitung

1: Entpackt die Datei

2: Startet nun LiveScanServer.exe und bei vorhandendensein einer Azure Kinect zusätzlich LiveScanClient.exe. Solltet ihr keine Kinect haben, startet ihr stattdessen LiveScanClient_Simulated.exe.

3: Klickt im LiveScanClient oben links auf das Plus, um so viele Azure Kinects hinzuzufügen wie ihr angeschlossen habt zu starten. Im simulierten Modus könnt ihr bis zu 4 virtuelle Kinects starten.

4: Klickt nun im LiveScanClient auf Connect All. Damit verbinden sich die Kameras mit dem Server

5: Im 3D-Viewport könnt ihr nun frei die Szene erkunden. Zum drehen der Ansicht, haltet die linke Maustaste gedrückt und bewegt die Maus. Um die Ansicht zu verschieben, haltet die rechte Maustaste gedrückt und bewegt die Maus. Mithilfe des Mausrads kann gezoomt werden.

6: Mithilfe des Start Capture Buttons könnt ihr eine Aufnahme starten. Diese wird im entpackten Ordner unter Out/NameDerAufnahme im .ply-Format gespeichert.

7: Probiert gerne die restlichen Einstellungen aus wenn ihr mögt! Bei den virtuellen Kameras können einige Einstellungen nicht geändert werden, insbesondere die Exposure und White Balance.

Danke fürs Testen!

Noch fragen oder etwas funktioniert nicht? Wendet euch gerne an mich über die Email-Adresse im Formular

With the start of 2024, and a new release of LiveScan3D coming closer and closer, I’d like to use this post to recapture what has been happening so far, where we are now, and most of all, what still needs to happen until the first big update of LiveScan3D.

For those who are unfamiliar with the project, LiveScan3D is a open-source tool for capturing low-cost volumetric video with multiple depth sensors, like the Kinect. Read more about the project here.

I’m writing this blog post partially to make the development process more transparent. But when I’m honest, the biggest reason is to set a fixed goal for myself and get my feature creep under control.

What happened so far

Originally, LiveScan3D was a software developed as a research project, and was meant to be mainly used in research aswell, but not for serious volumetric video production. My goal for the first big update is to bring LiveScan3D into a robust, usable and production-ready state, which anyone can use to produce their own volumetric video. Let’s start with a quick comparison of the last official release version of LiveScan3D and the current development build:

So, quite a bit has changed! Let’s start with the most visible part, the UI.

UI / UX

The UI in the old LiveScan version is a bit all over the place. A lot of windows and screen real estate is used, so I spent some time reducing and decluttering the UI. On the server app, which is the main windows with which you’ll be interacting, I merged the live 3D preview window with the control panel window, and sorted the buttons into a categoric layout. You’ll also notice a lot more buttons in general, but more on that later. On the client side, you now don’t need to open one app per camera anymore! You can just open it once, and then add as many cameras as you want with the new tab-based system.

Aesthetically speaking though, the UI will still be based on the relatively ugly windows forms for now. But behind the scenes, I’ve done a lot of work to untangle the application core from the UI code, so that replacing the UI framework should not be too much work.

Volumetric Capture

Seen from a bigger perspective, the capture & calibration process is still the same. You calibrate your cameras with printed markers, and LiveScan3D captures the volumetric data as point clouds. But there have been many improvements, which combined, do make a large difference in capture quality. Captures are now temporally synced between cameras and you can control the resolution, depth mode, exposure and white balance. Beside pointclouds, you can now capture the raw image output, which gives you much more options for post-processing.
Thanks to incremental improvements in the code, the application performance has been increased a lot. Instead of being able to run only two clients at 30 FPS on a Ryzen 3600, this is now up to four (Even more if you capture raw data).

Playback

Capturing volumetric video is nice, but also kind of useless if you can’t playback the recorded data. So I developed a open-source Unity3D package, creatively named “Unity Geometry Sequence Streaming“. With it, you can stream large pointcloud and mesh sequences from disk into Unity. It’s based on Unitys Mesh Jobs API, which makes it able to read and display geometric data very efficiently. Besides volumetric data, you can also use it to playback backed animations, like water and physics simulations, or animations with changing mesh topology. The plugin is already published and can be downloaded right now!

What still needs to be done

At the moment, LiveScan3D is still a bit of a loose stack of features, of which not all make sense to use, or are misplaced in the workflow pipeline. I’ve seen too many open-source projects which are in this state, which makes them confusing and frustrating to use and are ultimately kind of dropped by the community. Because of this, it’s important for me that LiveScan3D always represents a well-rounded and user-friendly experience. So here’s what I still want to implement before going for the release (If you want to, you can always check our Kanban board here, to see where we are exactly):

Untangeling the mess and tying up loose ends: The server and client applications is where almost all new features have been implemented, even though it doesn’t really make sense for most of them. These two applications should really only concentrate on configuring and reliantly capturing from the sensors. What happens with the data after that, e.g. any post processing, should be handled in another application, the LiveScan3D Editor.
Just as a few examples for the current bad state:

• The temporal synchronization process is happening directly after a capture, which is annoying as you have to wait until you can capture again. If something goes wrong, there is no way to sync your data again.
• If you want to capture pointcloud data, the capture applications have to convert, stitch and save the pointclouds, which gives bad performance, while you’re also loosing access to your raw data.
• If you capture raw data then…you can do nothing with it

So the plan is, to pull out all components on the server and client that are not needed for recording and put them into a volumetric video postprocessing software called LiveScan Editor. The pipeline for capturing a video should therefore be like:

1. Setup your sensors, calibrate and preview your capture space with LiveScan Server & Client
2. Capture only the raw data from the sensors with LiveScan Server & Client
3. Load your capture into the LiveScan Editor and postprocess
4. Export your capture into the desired format from the LiveScan Editor

Besides the improved user workflow, this gives other major advantages. The capture performance should be boosted by a large margin, as no processing needs to happen anymore at capture time. And as you retain all the raw data from the sensors, you can re-visit your captures later on, when there are more post-processing options released, and improve their quality.

Tests, Tests, Tests: LiveScan3D has not been the most stable software so far, crashes are frequent. This, of course, is bad in any software, but you especially don’t want your software to crash during a crucial moment in a capture you worked towards for days (Ask me how I know). We’re dogfooding LiveScan in the research lab I’m currently working on, but as I only need to use it occasionally, I don’t catch all of the bugs. Also, long bug-hunts and manual tests are not really an option for me, as I’m doing all of this in my spare time. So before the release, I want to write an automatic test suite, that covers most of the functionality. My preparations to separate the UI from the application has been especially shown to be useful here, as I can now test core functions, without having to interact go through the UI.
Of course, this alone won’t do it, so I hope that with some manual tests before the release and with the help of the community, I hope to achieve a stable version of LiveScan pretty soon enough.

Publication: When the software is finally read, the publication itself is also another work package on it’s own. Of course, I could go the open-source way, by writing a short how-to into the readme.md together with some incomplete build instructions and call it a day, but that’s not the way I’d like to go with LiveScan. From the start, I always wanted it to be a software that can be used by anyone, regardless of their technical skillset. So for this, reason, I’m creating a small static website right now, which contains everything you need to get started with LiveScan3D. This includes tutorials, documentation, downloads, videos and illustrations and is therefore some work in itself, which I’m hoping will pay off in the long run though! In my development experience so far, a well written documentation avoids a lot of issues and support needs. After the release, I’m also hoping to add a forum to the website, where the community can exchange on all topics regarding volumetric video!

Outro

If anybody read through all of this, thanks a lot!
I hope this gives you a better sense on where the livescan development cycle currently stands and what to expect from a first release. I hope to write more blog post, the closer we get to release and the more features are done! See you soon!

Read Article on ResearchGate
Download PDF

Abstract

Background:

Tying surgical knots is a basic but critical skill that surgeons must master. Currently, knot tying is typically taught through instructor observation or instructional videos. These methods are either very resource consuming or offer little interactivity and customizability. Knot tying training in extended reality (XR) could address these weaknesses and improve the linking of observation and application: For example, spatial awareness and the ability to mentally rotate, i.e. imagine knots from different perspectives, affects learning performance (Brandt & Davies, 2006). XR may support spatial awareness by allowing knot tying from different perspectives superimposed on the real world.

Method:

The project GreifbAR develops an XR-based interactive knot tying training application. This application teaches the process of knot tying and provides individualized feedback based on hand pose and scene recognition. To achieve a learning-friendly design and user acceptance, the requirements of learners and experts must be considered. Therefore, based on a literature review, an online survey with 80 medical students and four interviews with experienced surgeons at Charité – Universitätsmedizin Berlin were conducted.

Initial results:

The respondents show openness towards knot tying training with XR, yet emphasize the importance of a realistic learning situations and personal guidance. They also report little prior experience with XR. The talk integrates the survey results with findings from technology acceptance research and derives implications for the design of XR-based training systems for knot tying and similar procedural tasks.

LiveScan3D is an open-source, volumetric video capturing software. It was originally developed by Marek Kowalski and Jacek Naruniec and is now being maintained and by BuildingVolumes, a group trying to make volumetric video capture more widely available by developing open source tools for capture, processing and playback.

Do you want to help us build Livescan3D by testing our public beta? Sign up for the beta preview by writing us a short mail

Functionality

LiveScan3D uses multiple, low-cost RGBD cameras, such as the Kinect Azure, to capture a volumetric representation of the scene. It synchronizes all cameras spatially and temporarily, and then captures the scene into widely used file formats. You can record with up to 9 cameras. The distributed structure of LiveScan allows you to capture with a few low-end PCs, linked vial local network (Server and client structure) instead of one very high end PC, making the capture more economically feasible.

You can capture Pointclouds into the .PLY Format, or raw color & depth data in png. & .tiff frames for further post-processing.

LiveScan also includes a Playback-Application which lets you view your volumetric captures in interactive 3D and realtime, and experiment with the visualization.

Roadmap

At the moment, LiveScan3D is still in a pretty basic state and it is not ready to use in a production setting. The capture quality is low, UI/UX is is a rough shape, and there are many bugs

We’re aiming for a new release in mid 2023 that puts LiveScan3D on a solid base, with an improved workflow, tools and capture quality, aimed towards a production-ready state, that allows small content creators, artists and everybody else to produce volumetric videos.

The new version of LiveScan3D will feature the following improvements:

After this first release, we’re planning to add more features to close the gap to commerically available volumetric video capture software.

• Stable temporary synchronization of up to 9 cameras
• An improved spatial synchronization workflow
• An overhauled UI, for a more comfortable workflow
• Control of camera settings, such as whitebalance and exposure
• A better preview, in 2D as well as 3D
• Pointcloud and raw data capture, for further post-processing
• An improved video playback tool
• Playback in Unity3D

The next big step after the initial release is to offer a meshing & texturing tool, which allows you to capture your scene in much higher detail, and playback your capture in well-known tools such as Blender, Unity or Unreal Engine. This is scheduled for Q1 of 2025. You can follow the development of Livescan3D in my blog!

LiveScan3D is an open-source project that is developed by a very small team. If you share our vision and want to help accelerate development by contributing, testing, or managing, we’d be very happy to have you onboard!

Links

BuildingVolumes on Discord

BuildingVolumes on Github

KlimaVR is a Virtual Reality Experience which aims to teach school children about the causes, scale and consequences of climate change. This experience is still under development and will be released for free on the Oculus App Lab for all Quest devices and for PCVR later this year. KlimaVR aims to teach about the climate crisis in three segments:

The Mechanics

This chapter aims to visualise how the systems of climate change works and which role the different greenhouse-gases play. It explains the effects of CO² over- and underproduction on a global scale and which effects a rising CO²-level in the atmosphere has on our planet, and the ecosphere.

The Scale

This chapter aims to give an understanding for the scale of our overproduction of CO². How would CO² look like if we could see it? And how much do we produce ourselves, as a country, or as world population in a year? The journey will start with your personal footprint, but grow steadily in scale, over the borders of the city, as we try to visualize the CO² footprint of humankind since the industrial revolution. This is the level we’re still working on the most. Visualizing the sheer amount of CO² is a huge, but interesting challenge, not only on a technical level, but also from a design perspective.

The Consequences

In this chapter, you’ll be experiencing one of the many consequences of climate change, as you’re standing inside of a forest fire. It’s just a short glimpse into the many crisis climate change will bring with it, but it’s one of the most immersive parts in this app.

Funding

This project was funded by the Prototype Fund, a project of the Open Knowledge Foundation, financed by the Federal Ministry for Research and Education of the Federal Republic of Germany

Prototype Found

Concept

Surgical practice usually separates between imaging and intervention: i.e. a surgeon has to apply a CT scan to the patient’s anatomy cognitively. With this concept, we want to show how this cognitive gap can be briged with the use of Augmented Reality Headsets.

Phantom

A central part of our concept is the Phantom, a 3D-printed anatomic puppet, which is based on real patient data. Not only is the phantom geometry true to reality, the tissue is also true to touch, e.g. a tumor is harder than fatty tissue. This allows us to test Augmented Reality concepts on a surface which is very close to the actual conditions in the operating room. The phantom is fitted with a Marker, which allows us to detect it’s location and rotation in the real world with a Hololens 2 device.

Augmenting the liver

We used the Unity3D engine in combination with the Hololens 2, to augment the liver tissue with different experimental visualisations. The Hololens 2 detects the Phantoms presence by scanning a marker positioned on the bottom of it. You can then see different visualisations that are based on the same CT-Data as the phantom puppet. The visualisations are quite experimental and represent more of a testbed, to see which challenges might appear when augmenting human tissue and how our visual system interprets them.

One interesting learning we had is, that complete X-Ray Vision, e.g. being able to always see through the phantom body onto the augmented surfaces, is heavily distracting and an unpleasant experience. Therefore we created a mask, which lets you only see the augmented content when you also directly look into the abdominal cavity, the skin being the border for the content.

Open-Source Code

The Code is freely available on our Github. You need Unity 2019.2.21f1 (exactly this version!) and a Hololens 2 device to be able to play this project:

View the Code

VolumetricOR is the exploration of a concept, which allows surgical staff to learn and train surgical interventions in a photorealistic virtual operating room. Situated in the virtual reality environment, users can experience surgical workflows based on Volumetric Video recorded directly in the operating theater.

Photogrammetric Operating Room

For this project, I photographed and reconstructed an operating room with Agisoft Metashape, but due to the large amount of reflective, transparent and montone surfaces in this scan, I reconstructed most of the model by hand and then re-projected the textures onto the new surfaces in Metashape. You can use this model for free in your projects, download it on sketchfab:

Volumetric Video Capture inside of the OR

The biggest challenge for this project was capturing a volumetric video of a real kidney transplantation. We had to gurantee that our setup would not obstruct the surgical stuff in any way, which meant that we had to make the setup as small as possible. Each capture unit contained a Kinect V2 depth sensor in combination with a DSLR camera, and a mini desktop PC.

In total, three of these setups were used and connected via a local Wifi network to control them. The capture was then controlled via a seperate laptop outside of the operating room. We used DepthKit Pro and LiveScan3D as capture software, and then cleaned and imported the sequences into Unreal Engine. The quality of these captures is pretty rough, as you can see in the video at the top, or in this static frame below. This is a state from 2018 however, things improve quickly in this field.

Publication:

VolumetricOR: A new Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education

Queisner M, Pogorzhelskiy M, Remde C, Pratschke J,  Sauer IM, Surgical Innovation. 2022

Open Source Code

The complete project is availble as open-source project. Open the source files in Unreal Engine 4.19. You need a HTC-Vive, or a similar compatible SteamVR-Headset to be able to view the project in VR

Download