Microsoft has a trick for making holograms from live video

29 Jul 2015 | Author: | No comments yet »

Here’s How Microsoft Is Making 3D Videos For HoloLens.

We don’t know exactly what gaming is going to look like when it comes to Microsoft’s take on augmented reality, but it sure looks like the company could bring us the next generation of full-motion video (FMV) games with HoloLens.Microsoft researchers have published a paper detailing a technique to record live action ‘holographic’ video suitable for the company’s HoloLens augmented reality headset.

The company has released a new video that shows how it captures and translates real people and objects into 3D characters that work in the HoloLens head-mounted display. To create these “holograms,” Microsoft points more than 100 cameras at actors and then runs that video data through a number of algorithms to build the three-dimensional models.

While VR video capture focuses on placing users in a completely new scene, AR’s unique capability is combining the user’s real world with digital elements. Inserting a computer-generated object (say, a cube) in AR is simple enough as new views of the object can be computed on the fly as the user moves about. And once you see it in action, it’s not a huge leap to imagine game developers using similar tech to build games around recording video like this — just like studios used to back in the early days of CD gaming with FMV. All of these cameras take in information at various angles and compile it together so that they can create realistic three-dimensional models and spaces.

And unlike the process that was required to capture the performance of Gollum in Lord of the Rings, no special green screen suit is required by the performer. The abstract continues with wonderful technical jargon: Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format.

The cameras turn the action into a 3D point cloud, then algorithms further refine the model into tens of thousands of points, which is ultimately reduced to thousands of triangles per frame. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Although the HoloLens has been criticized for having a limited field of view, these incredibly detailed performances will undoubtedly change the way we consume entertainment in the future. Far away from the Redmond studio, astronauts on the International Space Station will eventually test HoloLens in hopes that they will soon be able to give people a firsthand look at what they’re seeing.

Although motion capture systems have been doing something similar for many years, most systems only capture the motion of performers which is then used to animate digital models. Areas of particular detail (like hands and faces) are kept at a higher level of quality than more simple areas as the entire mesh is brought down to a more manageable triangle count. Microsoft Corporation is a public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through … read more » The system looks quite capable, resulting in high quality output for a wide variety of captures including quickly moving objects, bright colors, reflections, and thin fabric, though it isn’t clear how much manual post-processing is needed to achieve such results.

The output from the technique should be equally applicable for VR headset which makes it a potentially attractive method for capturing live action performance for immersive displays.

Here you can write a commentary on the recording "Microsoft has a trick for making holograms from live video".

* Required fields
Our partners
Follow us
Contact us
Our contacts

ICQ: 423360519

About this site