Early Immersive Filmmaking Experiences for Vision Pro

Christoph Tilley from MXR Productions was interviewed for the CineD Focus Check podcast, and talked about Immersive filmmaking for Vision Pro:

This was a great conversation, during which Tilley talked about how hard it is to shoot Immersive content; summarised MXR Productions' process for creating high resolution, near-to-real-life Immersive video; as well as the challenges around distributing and playing Immersive content on Vision Pro – something Apple is yet to implement in visionOS outside of their own AppleTV app.

MXR has been shooting with both the Red Raptor and Canon's R5C or R5 MkII, both of which are capable of shooting 8k – the Canon in 60fps, and the Red in 120fps.

On their first Immersive commercial film project – for shoe company Asics – Tilley talks about how they faced the challenge of creating their own Immersive video player to be able to show their own content on the Vision Pro. To do this, they licensed code from James Hustler's Explore POV app and created their own player for visionOS.

Elaborating on the challenges with even just playing your own Immersive films on the Vision Pro at the moment [let alone creating them – I've included details of MXR's production process below], Tilley says:

"I think Apple might not be super interested in opening up [Immersive video playback] for third party generated content, because it's really hard to get it right and make it look good. If you open it up and the content is bad, it won't make the device look good."

...which I agree has to be the reason Apple hasn't done this – yet.

There's also the challenge of sharing Immersive videos with the public (or with a client's staff). How do you share Immersive videos with more than one other person?

Here, Tilley speaks about MXR's initial idea to have an "experience booth" at a public event – but they soon realised it would have been impossible for everyone to put on the Vision Pro and easily watch the videos – new users have to go through a calibration process which takes a minute or two, and they may not immediately know how to navigate the UI or play the video, etc.

To alleviate any "navigation" issues when showing other people their Immersive film, MXR intentionally made the "Play" button in their app "large and unmissable" – both visually, and with any possibly uncalibrated tap gestures by new users.

Immersive Film Production Process

Tilley also outlined MXR's production process for Immersive video at the moment, which – when written out even in just a handful of bullets – shows how challenging it currently is:

1. Shoot in 8k, 60fps (Canon) or 120fps (Red)
   1. Can't have anything like a dolly track in front of the camera, because it's in shot. Can't ever roll the camera. Can't move too quickly, etc.
2. Colour grade (but only to make it look as real as possible)
3. De-noise (because it needs to resemble what you see with your own eyes) – using Neat Video, in Resolve, on the 8k footage.
4. Export colour masters, which go into MistikaVR for convergence adjustments.
5. Upscaling using Topaz AI from the 8k colour master to 16k – "this takes a lot of time... we have some beefy computers with two 4090GTXs, which renders 0.5 frames per second. The upscale also takes away a bit more noise and improves detail a lot."
6. Mike Swanson's "spatial" tool is used to create the final MV-HEVC file for the Vision Pro.

Finally, I also found it interesting what Tilley said about his experiences owning the first "spatial computer" from Apple compared with the first iPhone, and showing both devices to friends and family who haven't/hadn't seen either device before...

"The most interesting similarity between the first iPhone and the Vision Pro for me is: I remember back in the day when I had my iPhone and showed friends the device, they were like 'woah, what is this?' They saw [multi-touch] pinching, a phone without buttons (...) and I have the same reactions and same experience when I show people the Vision Pro. (...) People put on the device and they try out the interface, they see the movies, they try out the environments, and everybody is blown away."
– 🕐 video timestamp 36:10