Apple Vision Pro: Why Entertainment Will Never Be The Same Again

Apple's upcoming augmented reality headset, the Vision Pro, promises to be a new kind of computing device, ushering in whole new ways of working. But while some of its features could be dismissed for being a bit fanciful, I believe what is worth getting excited about is how the Vision Pro will enable new ways of experiencing the world and what happens in it.

🍿 Watch this article! It was published as a video on my YouTube channel…

Thoughts? Comment over on the YouTube video (opens in new tab) 🙂

[When the device was first announced] Apple and Disney showed off a few concepts of what we might be able to experience on the Vision Pro, but I think these examples are misleading. This isn't what events experienced through the Vision Pro will be like. They'll be even better. And it's not just me that thinks so.

Honestly, I don't think the keynote did it justice. I've never had an experience like that with technology. My mind was blown and I feel like I haven't heard people talk about this yet. Maybe because they weren't, you know, filmmakers. But I think this is what's going to change filmmaking. Storytelling. YouTube content, everything. All I have to say is just wait till you experience it.

— Matti Haapoja, Filmmaker and YouTuber.

In [this article, and the above video] we'll take a look at a few of the leaps in technology that make the Vision Pro such a compelling device, capable of reshaping the way we experience everything from sporting events to concerts, remote destinations to our most personal memories.

Here's why entertainment – once the Vision Pro is released – will never be thought of or experienced in the same way again.

The Vision Pro, when it was announced by Apple, was described very intentionally as the company's first spatial computing device and, “the most advanced personal electronics device ever.” Apple highlighted some interesting use cases starting off by focusing on work related tasks such as group face time calls and shared whiteboard experiences and personal apps like web browsing and viewing photos and videos. Nothing too groundbreaking there, right?

Disney also took part in the announcement and showed some mockups, including watching The Mandalorian from a land speeder in the Star Wars universe. And then they showed us what watching basketball or American football might be like, bombarding us with as many 3D graphics as possible. If that seems jarring to you, perhaps it's because it was the complete opposite of Apple's focus with the Vision Pro.

In this clip [02:17 in video & pictured, below] Disney is attempting to bowl us over with a mind-numbing display that screams technology, but which turned me off completely.

Apple, in stark contrast, is going in the opposite direction. They're making the technology disappear in order to make the Vision Pro the most personal and compelling device they've ever made.

Let's take a look at what makes the Vision Pro truly remarkable.

The Vision Pro has a unique dual chip design with both an Apple silicon M2 processor and a new R1 processor. We currently know very little about the R1, except that it handles the processing of all of the device's sensors. It does this in real time, thanks to a real time operating system [RTOS], or subsystem, running on the R1 alongside the M2.

Such technology has been decades in the making. Back in 2016, Apple hired Dan Dodge, founder of QNX, who developed one of the first real time operating systems, but their work began in the 1980s. In the late 1990s, Commodore announced they'd struck a deal with QNX to use its neutrino operating system as the basis for the Amiga's next OS, before causing confusion the very next year when they announced they'd switched their focus to Linux.

But I digress.

Fast-forward back to the future, and here we are with Apple using a real time operating system in the upcoming Vision Pro. This is critical because it's allowed them to reduce the latency of the device, the time it takes from the cameras on the vision probe receiving information to that image being presented to your eyes to, “within 12 milliseconds.”

That's insanely fast. To attempt to draw some form of comparison, the passthrough latency of the Quest Pro headset has been measured to be between 35 and 60 milliseconds. If we average that to 48, the Vision Pro promises to be four times faster.

Having this real-time capability is vital because the Vision Pro is, by default, an augmented reality device almost always overlaying digital content on your environment rather than replacing it. It's effectively always operating in passthrough mode. The only thing you see when you first put on the headset is, well, nothing, just the world around you. No user interface appears at all until you summon it by pressing the digital crown.

The real time operating system running on the R1 helps ensure your reality is always displayed to you as it really is. So if one of your apps crashes your view of the world, the vision pro's passthrough should persist, allowing you to still see the world around you rather than reverting to a black screen.

Here's a compelling account of how this low latency makes the Vision Pro feel, courtesy of Myke Hurley, who described his demo experience with the Vision Pro on the Cortex podcast:

The biggest thing is the latency. It is imperceptible. There is a moment in the demo where they make me stand up and move around and I felt so comfortable to just get up and walk.

– Myke Hurley

The Vision Pro's insanely low latency really is a game changer for helping the headset feel comfortable to use, something which no other device has yet accomplished.

Apple then pairs this with the Vision Pro's insanely high resolution displays totaling over 23 million pixels. The pair of advanced micro-led panels are mounted seemingly on top of the M2 and R1 chips and manage to fit an eight-by-eight grid of 64 pixels into the space of a single iPhone pixel. Of course, you normally hold your phone further away from your face than a headset, so when you bring the display much closer, the individual pixels need to be much smaller to pull off the same disappearing trick.

Apple may have just achieved it.

By most accounts, the Vision Pro sets a new, virtually lifelike standard for displays and a headset. People who got to try it out have described the displays as virtually identical to what you'd see with the naked eye: No distortion, perhaps with slightly warmer colors. It sounds simple, but like the best referees at a football match, if you don't notice them, it means the displays are doing a great job.

Here's John Gruber talking about the fidelity of the Vision Pro's displays:

You know, you've got some goggles on, right. Yes, I've got goggles on. And I see that I’m looking through something. But what am I seeing? I'm just seeing the world around me as I would expect to see it. And yes, they've spent like billions of dollars, thousands of engineers, just to show you what you can see through a $5 pair of safety glasses.

– John Gruber

…and Jason Snell on This Week in Tech:

I'd say it's remarkable. And I say that you can very quickly forget that you're even looking at screens and it feels like you're looking at reality.

– Jason Snell

Other people who've got to try the Vision Pro have reported that it was perfectly possible to read and interact with a phone or watch while wearing the device, something you can't do comfortably with any other headset in passthrough mode. The result of pairing such high resolution displays with the Vision Pro's insanely fast pass-through and computing power is that Apple has essentially made the computer vanish.

Yes, the Vision Pro – or the device it ultimately evolves into – might well replace your laptop or desktop, perhaps even your phone and your watch, because it's an extremely powerful computer with a screen that's infinite and a display that's invisible.

The vanishing computer

The smartphone took the computer, the physical keyboard, and display – three separate devices – and combined them into one device you could fit in your pocket.

The iPhone replaced the hardware keyboard with software, enabled by the multitouch screen. Suddenly, the keyboard could be whatever it needed to be at any particular moment, or not there at all. Two devices were left, the computer and its display.

Now, the Vision Pro is making the display disappear. Rather than having a fixed physical screen, any number of virtual displays can be placed anywhere you like, resized and moved at will, or can quite literally vanish when not needed, leaving just you and your environment – or an imaginary one. As well as doing away with physical screens, thanks to the array of sensors they've built into the Vision Pro, in Apple's own words:

it's so precise it completely frees up your hands from needing clumsy hardware controllers.

– Mike Rockwell, Apple

That they've made the obvious choice of doing away with hand controllers is the cherry on the cake, or the final flourish in the trick of making technology essentially invisible and like magic.

So what will we be able to do with it?

The Vision Pro promises to be a spatial computing device in every aspect. Not only is it an extremely powerful computer that will be able to run thousands of existing Mac, iPhone and iPad apps; it will also allow us to place apps, windows or virtual objects in space anywhere around us.

We're not just talking about multiple virtual displays surrounding you at your desk where you'd otherwise put physical monitors or a giant cinema screen in place of your TV. You'll be able to summon interactive three dimensional things as well. Fancy putting a custom designed timer on your fridge or an infinity page recipe book on your kitchen counter? Download one and put it there. Want an animated photo frame on your living room wall? Load it, push it onto your wall and tell it which album to display photos from.

As you move between rooms, your digital things will stay wherever you place them, just like physical objects. Truly spatial computing.

So perhaps even the concept of apps – the user interface in a window that we interact with by tapping buttons – will disappear too, just like screens are about to, and hardware keyboards did before them. And instead of these apps we’ll have, simply, things.

Experience is Everything

The computing aspect of the device isn't what intrigues me the most, though. It's what we'll be able to experience through the Vision Pro that has me really excited for the future.

In 2023, the experiences you can have with a VR headset are technically interesting, but certainly don't fool you into thinking you're really there. They amount to relatively low resolution 180- or 360-degree videos of things like helicopter flights, musical performances, rollercoaster rides or walking tours.

The sound is sometimes good, but the pictures are grainy and the low resolution of today's headsets and the lack of real computing power to deliver features like spatial audio make for an unconvincing experience.

But with the Vision Pro, these experiences are set to take a monumental leap.

In 2020, Apple bought Next VR, a company that was developing technology for broadcasting sports and entertainment events in high definition stereoscopic video.

Apple has also reportedly developed camera systems capable of capturing Apple immersive video specifically for experiencing on the Vision Pro and its successors. If you can imagine what the evolution – or perhaps revolution – of these broadcast experiences will be like, with high fidelity video and spatial audio, it's genuinely going to change how we'll be able to enjoy sporting events, concerts, remote locations and even our own memories.

It's with the latter that Apple will perhaps hook us all in the most. It's easy to imagine an upcoming iPhone having dual cameras to capture stereoscopic video ready to relive with the Vision Pro. [Apple later announced that spatial video recording was going to be capable on the iPhone 15 Pro]. Imagine the camera technology Apple's developed being deployed in remote locations or at major events, recording truly high resolution immersive video and audio to be delivered through the Vision Pro.

We’ll feel like we're really in that remote location or that we're really at the event, even being able to view them from places we'd never be able to position ourselves in reality, like above the basketball hoop or goalposts, on top of the stadium or on stage alongside the performers. Suddenly with this technology, we'll all be able to be there, and these experiences will feel almost indistinguishable from reality.

With this technology – like experiences you could only have with a 4K TV, an ultra high definition Blu-Ray player in the mid 2010’s, the mid 2020’s promise to deliver 3D experiences – thanks to the Vision Pro and the technology currently behind closed doors – with the visual fidelity and spatial audio quality needed to make them feel truly real.

Given the choice of purchasing a 4K Blu-Ray or buying the virtual experience, a lot of people will be picking the virtual experience.

I know I will, and I can't wait.