With the start of 2024, and a new release of LiveScan3D coming closer and closer, I’d like to use this post to recapture what has been happening so far, where we are now, and most of all, what still needs to happen until the first big update of LiveScan3D.
👉 For those who are unfamiliar with the project, LiveScan3D is a open-source tool for capturing low-cost volumetric video with multiple depth sensors, like the Kinect. Read more about the project here.
I’m writing this blog post partially to make the development process more transparent. But when I’m honest, the biggest reason is to set a fixed goal for myself and get my feature creep under control.
What happened so far
Originally, LiveScan3D was a software developed as a research project, and was meant to be mainly used in research aswell, but not for serious volumetric video production. My goal for the first big update is to bring LiveScan3D into a robust, usable and production-ready state, which anyone can use to produce their own volumetric video. Let’s start with a quick comparison of the last official release version of LiveScan3D and the current development build:
So, quite a bit has changed! Let’s start with the most visible part, the UI.
UI / UX
The UI in the old LiveScan version is a bit all over the place. A lot of windows and screen real estate is used, so I spent some time reducing and decluttering the UI. On the server app, which is the main windows with which you’ll be interacting, I merged the live 3D preview window with the control panel window, and sorted the buttons into a categoric layout. You’ll also notice a lot more buttons in general, but more on that later. On the client side, you now don’t need to open one app per camera anymore! You can just open it once, and then add as many cameras as you want with the new tab-based system.
Aesthetically speaking though, the UI will still be based on the relatively ugly windows forms for now. But behind the scenes, I’ve done a lot of work to untangle the application core from the UI code, so that replacing the UI framework should not be too much work.
Volumetric Capture
Seen from a bigger perspective, the capture & calibration process is still the same. You calibrate your cameras with printed markers, and LiveScan3D captures the volumetric data as point clouds. But there have been many improvements, which combined, do make a large difference in capture quality. Captures are now temporally synced between cameras and you can control the resolution, depth mode, exposure and white balance. Beside pointclouds, you can now capture the raw image output, which gives you much more options for post-processing.
Thanks to incremental improvements in the code, the application performance has been increased a lot. Instead of being able to run only two clients at 30 FPS on a Ryzen 3600, this is now up to four (Even more if you capture raw data).
Playback
Capturing volumetric video is nice, but also kind of useless if you can’t playback the recorded data. So I developed a open-source Unity3D package, creatively named “Unity Geometry Sequence Streaming“
With it, you can stream large pointcloud and mesh sequences from disk into Unity. It’s based on Unitys Mesh Jobs API, which makes it able to read and display geometric data very efficiently. Besides volumetric data, you can also use it to playback backed animations, like water and physics simulations, or animations with changing mesh topology. The plugin is already published and can be downloaded right now!
What still needs to be done
At the moment, LiveScan3D is still a bit of a loose stack of features, of which not all make sense to use, or are misplaced in the workflow pipeline. I’ve seen too many open-source projects which are in this state, which makes them confusing and frustrating to use and are ultimately kind of dropped by the community. Because of this, it’s important for me that LiveScan3D always represents a well-rounded and user-friendly experience. So here’s what I still want to implement before going for the release:
(If you want to, you can always check our Kanban board here, to see where we are exactly).
Untangeling the mess and tying up loose ends
The server and client applications is where almost all new features have been implemented, even though it doesn’t really make sense for most of them. These two applications should really only concentrate on configuring and reliantly capturing from the sensors. What happens with the data after that, e.g. any post processing, should be handled in another application, the LiveScan3D Editor.
Just as a few examples for the current bad state:
- The temporal synchronization process is happening directly after a capture, which is annoying as you have to wait until you can capture again. If something goes wrong, there is no way to sync your data again.
- If you want to capture pointcloud data, the capture applications have to convert, stitch and save the pointclouds, which gives bad performance, while you’re also loosing access to your raw data.
- If you capture raw data then…you can do nothing with it
So the plan is, to pull out all components on the server and client that are not needed for recording and put them into a volumetric video postprocessing software called LiveScan Editor. The pipeline for capturing a video should therefore be like:
- Setup your sensors, calibrate and preview your capture space with LiveScan Server & Client
- Capture only the raw data from the sensors with LiveScan Server & Client
- Load your capture into the LiveScan Editor and postprocess
- Export your capture into the desired format from the LiveScan Editor
Besides the improved user workflow, this gives other major advantages. The capture performance should be boosted by a large margin, as no processing needs to happen anymore at capture time. And as you retain all the raw data from the sensors, you can re-visit your captures later on, when there are more post-processing options released, and improve their quality.
Tests, Tests, Tests
LiveScan3D has not been the most stable software so far, crashes are frequent. This, of course, is bad in any software, but you especially don’t want your software to crash during a crucial moment in a capture you worked towards for days (Ask me how I know). We’re dogfooding LiveScan in the research lab I’m currently working on, but as I only need to use it occasionally, I don’t catch all of the bugs. Also, long bug-hunts and manual tests are not really an option for me, as I’m doing all of this in my spare time. So before the release, I want to write an automatic test suite, that covers most of the functionality. My preparations to separate the UI from the application has been especially shown to be useful here, as I can now test core functions, without having to interact go through the UI.
Of course, this alone won’t do it, so I hope that with some manual tests before the release and with the help of the community, I hope to achieve a stable version of LiveScan pretty soon enough.
Publication
When the software is finally read, the publication itself is also another work package on it’s own. Of course, I could go the open-source way, by writing a short how-to into the readme.md together with some incomplete build instructions and call it a day, but that’s not the way I’d like to go with LiveScan. From the start, I always wanted it to be a software that can be used by anyone, regardless of their technical skillset. So for this, reason, I’m creating a small static website right now, which contains everything you need to get started with LiveScan3D. This includes tutorials, documentation, downloads, videos and illustrations and is therefore some work in itself, which I’m hoping will pay off in the long run though! In my development experience so far, a well written documentation avoids a lot of issues and support needs. After the release, I’m also hoping to add a forum to the website, where the community can exchange on all topics regarding volumetric video!
Outro
If anybody read through all of this, thanks a lot! 🙂
I hope this gives you a better sense on where the livescan development cycle currently stands and what to expect from a first release. I hope to write more blog post, the closer we get to release and the more features are done! See you soon!