Android XR to Feature Phone-Like Camera Access

The current buzz in the XR community centers around passthrough camera access, and while we’ve seen how giants like Meta, Apple, and Pico approach this, there’s been a big question mark looming over Google’s plans for Android XR. After a recent conversation with the company, I believe I can shed some light on this. They’re likely to channel a similar solution to what’s been implemented on phones. Want to know more? Keep reading!

The Dilemma Over Camera Access

Let’s rewind for a moment. For those new to the topic, it’s essential to understand that the latest standalone VR devices are more than just virtual—they’re actually mixed reality (MR) headsets. They utilize RGB passthrough from front-facing cameras to let users interact with the world around them through applications like Cubism and Starship Home.

The OS captures these frames to provide that immersive passthrough vision. However, developers are eagerly waiting for access to these camera frames as well. Why? With the ability to analyze user surroundings through AI and computer vision, reality could be enhanced way beyond its current limits. Imagine apps that are truly aware of their environment. For example, I once used a clever workaround on Quest to prototype an AI-powered MR app for interior design. Such innovations are just a glimpse of the potential that camera access holds.

Yet, this potential brings up a significant privacy debate. If dubious developers gain access to these cameras, they might exploit personal information. This could range from scanning IDs and bank cards left on tables to even capturing faces and bodies with malicious intent.

So, here we stand at a crossroads. On one hand, we need to protect user privacy; on the other, tapping into the full potential of mixed reality seems crucial.

The Approach of XR Firms

In the early days, headsets offered unrestricted camera access. If you’ve been following my journey, you might remember when my team at NTW experimented with technologies like diminished reality back in 2019. It was an era of exploration with full access to camera textures.

But as MR technology gained traction, companies, fearing privacy issues, tightened the reins. Meta, Pico, HTC, and eventually Apple, all shut down direct camera access for developers.

While this was initially accepted, the XR developer community soon felt restricted and began advocating vigorously for camera access. Key voices like Cix Liv, Michael Gschwandtner, and I called for a transparent system. If users allowed it, we wanted the liberty to introduce features like object recognition using what their cameras captured. We posed the question—why was XR treated so differently when phones granted camera access with just a prompt?

This movement bore fruit when Meta announced an upcoming "Passthrough API." But what about Google’s stance with Android XR?

Android XR – A Phone-Like Experience

Android, the operating system most smartphones worldwide run on, lets developers use camera streams once users grant permission. Assign the camera ID, and you can capture and analyze to your heart’s content.

In efforts to maintain compatibility, Google is expected to implement similar features with Android XR. Following several rumors and a direct email thread I had with a Google representative, here’s the essence of their plan:

Developers will be able to harness existing camera frames with user approval, just like on any Android app. Requests can be made for both the rear and selfie-camera streams.

This means XR developers can rely on tools like CameraX, which they already use on phones, to tap into camera feeds on XR headsets as well. Exciting news, right?

There is, however, a slight twist. Although the front camera directly shows the user’s avatar, the rear camera offers a reconstructed image, echoing Apple’s strategy with Vision Pro. This serves to standardize user experience—what the app captures while using "rear camera" or "selfie camera" aligns with traditional phone behavior.

A Word of Caution Regarding Android XR

It’s crucial to remember that Android XR is still under development with no official headsets released. The current strategies and features might undergo changes by the time they’re commercially available. While I don’t anticipate massive shifts, it’s worth considering this factor.

Embracing Camera Access

With entities like Google and Meta warming up to the idea, it’s likely that by 2025, we could witness a renaissance of mixed reality opportunities as more firms follow suit. I, for one, am eager to see the groundbreaking applications our community of developers will bring to life!

(Header image courtesy of Samsung)

Note: This blog contains ads and affiliate links to support our work. Clicking on them might earn me a small commission at no cost to you. Full disclosure available here.