Apple Vision Pro and the pillars of presence: things just got weird

With typical marketing contrariness, Apple avoided terms like “virtual reality” (VR) or “augmented reality” (AR) when launching the Vision Pro. Instead, it prefers “spatial computing”.

But the Vision Pro is, at its heart, a clever combination of VR and AR. It has a dial that allows you to choose how much you want to immerse yourself in a virtual world, like a dimmer switch between fantasy and reality.

Turn the reality dimmer one way to block out the real world and immerse yourself in a virtual reality experience. Or twist the dial and bring yourself back into your actual surroundings, albeit augmented with digital overlays you can control by waving your hands around.

For most virtual reality headsets and experiences, the challenge is making you feel like you’re actually in their VR fake worlds. When VR successfully fools you, you get those magic/terrifying moments.

The Vision Pro, on the other hand, must do lots of clever things to fool you that – when you turn that dial to AR – you’re once again interacting with reality. Which is where it gets odd.

That sense of being somewhere else, that strange and magical sense of senses fooled, is called “presence”.  And that’s what this article is all about.

The four pillars of presence 

There are a few elements that go into creating a sense of presence. Sound is a powerful driver of the illusion, as sounds that appear to come from different directions fool your brain into thinking you’re in a three-dimensional world.

Realistic visuals also give a big boost to presence, and the Vision Pro is hard to beat on that score. The powerful R1 and M2 chips fuel a high-resolution 4K screen for each eye, beaming in graphics a far cry from blocky VR graphics of the past. 

Buying that you’re in a different reality is more than just believing your eyes, however. In recent years, VR expert and creator Caecilia Charbonnier identified what she termed four pillars of presence.

Charbonnier argued the technology must create an illusion of being in a stable space, for which graphics play a major role. Presence is further enhanced if the technology makes you feel and even see your body move in the virtual world. The most powerful components of presence involve interacting with the pretend world: physically, with controllers or other objects, and, finally, with other people in the virtual space.

Along with realistic graphics, the illusion of self-embodiment is something that the Vision Pro is already very good at. Small but powerful cameras point downwards from the bottom of the visor to track your movements, almost acting as peripheral vision.

It’s the last two pillars of presence – convincing physical and social interaction – that eluded previous generations of VR. Again, Apple appears to have cracked it: Gizmodo’s Dan Ackerman notes that the Vision Pro’s hand (and eye) tracking are “miles beyond” devices like the Oculus Quest.

So it’s a good VR headset. It’s the other stuff that’s weird.

Switching realities

The Vision Pro doesn’t just make you feel like you’re really in a digital world. It also works to make digital elements feel present in the real world.

Switch to augmented reality, and you see the actual real-life room you’re in – except with digital elements overlaid in front of your eyes. You can look at your living room and picture a TV that isn’t actually there, hanging in front of your eyes but not visible to anyone else. Or iMessage hovers in the air, and you can grab photos out of message threads. To sell the feeling that these digital elements are part of the real world, they cast shadows on the ground and react to the actual room’s lighting conditions.

In terms of presence, these AR versions of existing Apple apps have the advantage of familiarity. When first entering most VR experiences you have to orient yourself in a strange and often fantastical world. But when you strap on the Vision Pro, you’re in your living room looking at iMessage. Even if it’s your first time donning Apple’s headset, you know what that’s supposed to look like. 

The thing is, if you want to sit in your living room looking at iMessage… you can just do that. I’m going to take a break from writing this article and do it right now. I don’t need a huge pair of goggles when I can just get my phone or laptop out.

If I want to watch TV in my living room I don’t need to spend $3,499, the price of the Vision Pro, and I don’t need to strap a highly sophisticated piece of technology to my face to convince me I’m in my living room.

The most difficult thing about watching TV is finding the remote. Why complicate it further? 

Passthrough to the other side 

If you wanted to stand in your kitchen and talk to your roommate or spouse or child, you could take the headset off. Or you could turn the digital dimmer all the way to “the real world”, and you’ll see your kitchen in the goggles. This involves something called “passthrough”: the Vision Pro films the person in front of you, and plays the video back to you in the tiny screens in front of your eyes. 

In other words, you think you’re looking at the real person standing in front of you, but you’re not.

You’re looking at a recording of them, a digital recreation, a virtual ghost a fraction of a millisecond behind. It’s sort of like looking at the night sky stars and seeing what the stars looked like thousands of years ago. Or more accurately, it’s like having an actual star in your kitchen that you could just look at, but instead you choose to put on a hat with a TV in it and watch a recording of what’s happening three feet in front of your face.

That’s odd, right?

Face to face 

There are situations when this stuff makes sense, of course. These situations – what the experts call “compelling use cases” – include having a meeting with someone who isn’t standing three feet away from you. You can talk face to face with a colleague or loved one in another office or another country, and this time the illusion of presence is about creating the illusion of their presence.

But again. If you want to talk face-to-face meeting with someone in another place, you can do that. Get out your phone and FaceTime.

And even a good old-fashioned glitchy Zoom call with a colleague who keeps forgetting they’re muted has an advantage over Apple’s version of video calling. When you FaceTime with the Vision Pro, you’re not actually seeing video of the person you’re talking to. You’re seeing a “Persona”: a 3D face scan of the other person, like an avatar or profile picture, but animated so its mouth and eyes move when they talk.

Certainly it’s light years better than the embarrassingly blocky, legless avatars of Mark Zuckerberg’s now-defunct Metaverse. But it’s still weird.

It’s not just me, is it?

Erecting barriers

Apple will no doubt argue that the Vision Pro connects people. But I’m unsettled by the prospect of a device that erects even more digitally mediated barriers between us. Of course, it’s early days for this version of VR and AR, and perhaps I just lack the vision to understand how it will benefit us in future.

In fact, I can see a possible compelling use case. For a century, sci-fi movies promised that video calls would be ubiquitous in our future, and we had the technology for decades but it never took off. Until a pandemic physically separated us from colleagues, friends and loved ones. Suddenly, video calling became a lifeline, a way to interact with those we loved who couldn’t join us wherever we were.  

So when the next semi-apocalypse happens and we’re all hideously deformed mutants sheathed in all-encompassing exosuits, maybe we’ll be grateful to be able to stand in front of each other and see a 3D avatar smiling and laughing. 

The people who’ve tried the Vision Pro have called it “incredibly impressive”, “heavy”, and an “alarming misfire”. It’s an expensive and complicated way to do a bunch of things you can already do, but it appears to be better than most previous virtual and augmented reality devices.

So Apple can sell the illusion. The question is, who’s buying it?