Quickstart | Documentation | Unity Asset Store | Unity Forums Thread | License

The Geometry Sequence Player is a package for the Unity game engine which enables playback of large geometry sequences inside of Unity. The package can be used to playback pointcloud, mesh or textured mesh sequences.

It is available on either Github (for non-commercial use) or the Unity Asset Store (for commercial use):

What is a geometry sequence?

In a geometry sequence, each frame consists out of an individual mesh or pointcloud, which is shown at short Intervalls to create the illusion of an animation, kind of like a three dimensional flipbook. As each frame is independent from the next one, no limitations apply to your animation, unlike for example skinned meshes or blendshapes, where mesh topology can’t be changed. This applies for example to:

Pre-Rendered fluid caches
Baked Physiscs simulations
Animated Procedural meshes
4D scans with per frame textures
Volumetric videos
Pointclouds captured by RGBD sensors (Kinect, Realsense, Lidar)
Data visualisations

Features

• Can import sequences from almost any source, with an included converter that covers most standard file formats.
• Easy setup of sequences with an easy-to-use Editor UI and playback controls
• Integrated into the Unity Timeline
• Granular playback control for interactive experiences is provided through a scripting API
• Sequences don’t need to be loaded into your Unity project or even the RAM, everything is streamed at runtime from disk
• Highly efficient playback system that allows you to stream millions of points/polygons per second, even on lower end hardware

The package

Originally, this package was created to facilitate the playback of volumetric videos from LiveScan3D, captured with Depth sensors such as the Azure Kinect or Intel realsense. But I realized that it could also be useful for many more applications, such as getting fluid and physics simulations from Blender into Unity!

I have used the package intensively in many projects myself, so it’s well proven and tested. The package has been released on Github as open-source code for quite a while already, but to make the development and support a bit more sustainable, I now decided to also offer the package on the Unity Asset Store.
If you want to use the package commercially, or if you want to support our development, you can buy it there! Thank you very much for your support

LiveScan3D is an open-source, volumetric video capturing software. It was originally developed by Marek Kowalski and Jacek Naruniec and is now being maintained and by BuildingVolumes, a group trying to make volumetric video capture more widely available by developing open source tools for capture, processing and playback.

Do you want to help us build Livescan3D by testing our public beta? Sign up for the beta preview by writing us a short mail

Functionality

LiveScan3D uses multiple, low-cost RGBD cameras, such as the Kinect Azure, to capture a volumetric representation of the scene. It synchronizes all cameras spatially and temporarily, and then captures the scene into widely used file formats. You can record with up to 9 cameras. The distributed structure of LiveScan allows you to capture with a few low-end PCs, linked vial local network (Server and client structure) instead of one very high end PC, making the capture more economically feasible.

You can capture Pointclouds into the .PLY Format, or raw color & depth data in png. & .tiff frames for further post-processing.

LiveScan also includes a Playback-Application which lets you view your volumetric captures in interactive 3D and realtime, and experiment with the visualization.

Roadmap

At the moment, LiveScan3D is still in a pretty basic state and it is not ready to use in a production setting. The capture quality is low, UI/UX is is a rough shape, and there are many bugs

We’re aiming for a new release in mid 2023 that puts LiveScan3D on a solid base, with an improved workflow, tools and capture quality, aimed towards a production-ready state, that allows small content creators, artists and everybody else to produce volumetric videos.

The new version of LiveScan3D will feature the following improvements:

After this first release, we’re planning to add more features to close the gap to commerically available volumetric video capture software.

• Stable temporary synchronization of up to 9 cameras
• An improved spatial synchronization workflow
• An overhauled UI, for a more comfortable workflow
• Control of camera settings, such as whitebalance and exposure
• A better preview, in 2D as well as 3D
• Pointcloud and raw data capture, for further post-processing
• An improved video playback tool
• Playback in Unity3D

The next big step after the initial release is to offer a meshing & texturing tool, which allows you to capture your scene in much higher detail, and playback your capture in well-known tools such as Blender, Unity or Unreal Engine. This is scheduled for Q1 of 2025. You can follow the development of Livescan3D in my blog!

LiveScan3D is an open-source project that is developed by a very small team. If you share our vision and want to help accelerate development by contributing, testing, or managing, we’d be very happy to have you onboard!

Links

BuildingVolumes on Discord

BuildingVolumes on Github

KlimaVR is a Virtual Reality Experience which aims to teach school children about the causes, scale and consequences of climate change. This experience is still under development and will be released for free on the Oculus App Lab for all Quest devices and for PCVR later this year. KlimaVR aims to teach about the climate crisis in three segments:

The Mechanics

This chapter aims to visualise how the systems of climate change works and which role the different greenhouse-gases play. It explains the effects of CO² over- and underproduction on a global scale and which effects a rising CO²-level in the atmosphere has on our planet, and the ecosphere.

The Scale

This chapter aims to give an understanding for the scale of our overproduction of CO². How would CO² look like if we could see it? And how much do we produce ourselves, as a country, or as world population in a year? The journey will start with your personal footprint, but grow steadily in scale, over the borders of the city, as we try to visualize the CO² footprint of humankind since the industrial revolution. This is the level we’re still working on the most. Visualizing the sheer amount of CO² is a huge, but interesting challenge, not only on a technical level, but also from a design perspective.

The Consequences

In this chapter, you’ll be experiencing one of the many consequences of climate change, as you’re standing inside of a forest fire. It’s just a short glimpse into the many crisis climate change will bring with it, but it’s one of the most immersive parts in this app.

Funding

This project was funded by the Prototype Fund, a project of the Open Knowledge Foundation, financed by the Federal Ministry for Research and Education of the Federal Republic of Germany

Prototype Found

Concept

Surgical practice usually separates between imaging and intervention: i.e. a surgeon has to apply a CT scan to the patient’s anatomy cognitively. With this concept, we want to show how this cognitive gap can be briged with the use of Augmented Reality Headsets.

Phantom

A central part of our concept is the Phantom, a 3D-printed anatomic puppet, which is based on real patient data. Not only is the phantom geometry true to reality, the tissue is also true to touch, e.g. a tumor is harder than fatty tissue. This allows us to test Augmented Reality concepts on a surface which is very close to the actual conditions in the operating room. The phantom is fitted with a Marker, which allows us to detect it’s location and rotation in the real world with a Hololens 2 device.

Augmenting the liver

We used the Unity3D engine in combination with the Hololens 2, to augment the liver tissue with different experimental visualisations. The Hololens 2 detects the Phantoms presence by scanning a marker positioned on the bottom of it. You can then see different visualisations that are based on the same CT-Data as the phantom puppet. The visualisations are quite experimental and represent more of a testbed, to see which challenges might appear when augmenting human tissue and how our visual system interprets them.

One interesting learning we had is, that complete X-Ray Vision, e.g. being able to always see through the phantom body onto the augmented surfaces, is heavily distracting and an unpleasant experience. Therefore we created a mask, which lets you only see the augmented content when you also directly look into the abdominal cavity, the skin being the border for the content.

Open-Source Code

The Code is freely available on our Github. You need Unity 2019.2.21f1 (exactly this version!) and a Hololens 2 device to be able to play this project:

View the Code

VolumetricOR is the exploration of a concept, which allows surgical staff to learn and train surgical interventions in a photorealistic virtual operating room. Situated in the virtual reality environment, users can experience surgical workflows based on Volumetric Video recorded directly in the operating theater.

Photogrammetric Operating Room

For this project, I photographed and reconstructed an operating room with Agisoft Metashape, but due to the large amount of reflective, transparent and montone surfaces in this scan, I reconstructed most of the model by hand and then re-projected the textures onto the new surfaces in Metashape. You can use this model for free in your projects, download it on sketchfab:

Volumetric Video Capture inside of the OR

The biggest challenge for this project was capturing a volumetric video of a real kidney transplantation. We had to gurantee that our setup would not obstruct the surgical stuff in any way, which meant that we had to make the setup as small as possible. Each capture unit contained a Kinect V2 depth sensor in combination with a DSLR camera, and a mini desktop PC.

In total, three of these setups were used and connected via a local Wifi network to control them. The capture was then controlled via a seperate laptop outside of the operating room. We used DepthKit Pro and LiveScan3D as capture software, and then cleaned and imported the sequences into Unreal Engine. The quality of these captures is pretty rough, as you can see in the video at the top, or in this static frame below. This is a state from 2018 however, things improve quickly in this field.

Publication:

VolumetricOR: A new Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education

Queisner M, Pogorzhelskiy M, Remde C, Pratschke J,  Sauer IM, Surgical Innovation. 2022

Open Source Code

The complete project is availble as open-source project. Open the source files in Unreal Engine 4.19. You need a HTC-Vive, or a similar compatible SteamVR-Headset to be able to view the project in VR

Download

Dein Erster Tag

“Dein Erster Tag” (“Your first day”) is an educational program for students in Germany. The idea here is to let students immersively experience the jobs with 360° as if they would spend their first day there. In this way, the app can support students in their journey to find the right job, traineeship or study program You can take a look at the 360° videos on their Youtube Channel here.

LoomVR

LoomVR is an interactive 360° video application, in which users can safely experience difficult and stressful situations on their job site, for example menacing customers. The app combines realistic 360° 3D videos with interactive choices, and feedback by a virtual trainer to each decision.

My contributions

Together in a small team with Michael Pogorzhelskiy, I developed the Virtual Reality Applications for both projects from the ground up. While Michael took over the UI and UX, as well as design, I implemented the apps in the Unity3D engine. This covered all aspects of the app like menus, video personalisation, media playback and user statistics collection.
For LoomVR, I also developed a user-friendly node storytelling tool, which lets users, without any previous experience with the Unity3D engine, create interactive 360-video experiences.

Reception

The Dein Erster Tag app has and is being used by hundreds of schools and thousands of users across Germany each year. Teachers and schools can either rent a VR media box for free, or purchase it, which includes three VR headsets containing the Dein Erster Tag App, and instructional materials for schools lessons. LoomVR has been expanded to include over ten scenarios, and has been localized into 4 countries.