In my office, there are bookshelves where I keep my magic. Next to them are panoramic windows overlooking Paris — outside, not really. It’s the real hybrid overlays that greet me when I wear it Apple Vision Pro headset, mapped to my original windows. A virtual clock hangs on the wall.
Bookshelves are real; the paris windows and wall clock are missing. But Siri, whom I call as a glowing orb before me, sees them all. When I ask what’s in front of me, Siri’s text box explains everything.
My first few moments wearing the Vision Pro and testing the all-seeing Siri AI in the developer preview of the VisionOS 27 they are surprising but very familiar. I have had moments like this Samsung Galaxy XR headset, which already has a Gemini Live mode that can see your room and visible apps and open windows. I did it too he wears a lot of smart glasses where I asked a camera-aware AI to tell me about the world as I walked around.
As I expectedApple has the supercharged new Siri is coming to a variety of devices this fall, from Watches to iPhones to iPads, and is expanding its artificial intelligence capabilities beyond the iPhone. The Vision Pro’s Siri is the most exciting, though, and perhaps the most ambitious. It can see everything in your field of view on command, like a sensory companion.
Not everything is perfect in this early preview build, revealed a few days ago by WWDC for Applebut it is already working well enough to indicate what may follow. Yes, glasses that can do this sound totally possible now. Meanwhile, Vision Pro’s auxiliary functions will be very interesting, too.
Siri knows my bookshelf, and my virtual Paris window, and my virtual clock.
Siri’s glowing orb is ready to be seen from wherever you go
Unlike Visual Intelligence Modes on iOS and iPadOS, which launch through the Camera app, Vision Pro’s Siri can see things if you just say, “Hey, Siri,” and ask a question.
Siri appears as a glowing 3D orb, appearing in my room like a ghost. You can drag and drop it anywhere, but it’s nice to see the marble glow on my desktop or in the room using the local VisionOS graphics engine.
I was able to say, “Hey, Siri, what’s in front of me?” and trigger Visual Intelligence, making new chirps and flashing a quick visual scan of the area my eyes are looking at, using eye tracking. It casts a wide net: It can see my bookshelf full of books in front of me and all the little action figures and toys I put there. Read a few book titles back to me, like Uzumaki and Wonderbook.
The AI feedback comes after the camera captures a still image of what’s in front of me, real or virtual. It’s the same snap-response cadence as the other AI-based camera on the Meta glasses or the Samsung Galaxy XR.
But it doesn’t have a live mode like those glasses and headsets do. It is one-per-application.
My cluttered bookshelf is visible to Siri. Siri’s orb glows next to real-world objects, too, and can be pinched and dragged.
Real and visible awareness
On my desk, Siri was able to detect a red virtual reality headset (A Real Boy) and a The Steam Deckboth original. After that, I tried looking at my virtual Paris windows and wall clock widget and my magical bookshelf, and it saw it too.
In this early beta, Siri would often stick to a particular view for a while and analyze that capture without me having to turn off Siri or move the orb to try again. Or, I guess, I’m getting used to it.
But of course I wonder how I can use this visual awareness to catch up on work quickly. I asked Siri to summarize the notes app about my recent playwriting, which I pulled up just by asking (it summarizes the important points), and I asked it through the windows browser I had open on my MacBook via the virtual display. (It tells me right now, as I’m working on this story in Vision Pro on my Mac, that I have a Google Docs window open where I’m writing about Siri seeing real and virtual objects.)
I’m curious to know what Siri thought of my Virtual Boy and Steam Deck.
Converting my photos into 3D panoramic backgrounds
I also tried out the new panoramic image conversion effect in VisionOS 27, which allows any panoramic image in your library to turn into a 3D foldable background that you can use while you work.
The effects aren’t as full 3D as Apple’s immersive environments, and they don’t move or have surround sounds. They are like 3D windows that cover a lot, some of my offices are still visible at the edges.
It looks great, even if not all my panoramic photos are converted (it’s early days; I’m sure it will work). I drew a picture of my mother in her yard when I visited her during the plague, and now it greets me as if I were there again. I’d love it if Vision Pro added focused Gaussian splat images from multiple images, like I did in Meta Quest.
I can’t help but think how Apple’s expected smart glasses will use Siri’s vision for assistance purposes. But I’m moving forward. Right now, you need the $3,499 Vision Pro to make this happen. But when Apple has AR glasses like Google and Xreal’s Project Aura are they small and affordable? And if other apps can fit into the Siri framework? It will be interesting to see what happens next. I feel like I’m on the edge of things to come.
