How does light reach your eyes in the proper perspective in see-through augmented reality? The Mixed Reality headset from Meta has a special solution.
The Quest Pro and Apple Vision Pro mixed reality headsets have a drawback. A few centimetres in front of your eyes are where their outside cameras are located. As a result, the proper viewpoint is not captured of the outside world improved by computer graphics. The image must be projected in order for algorithms to rectify the deviation.
Meta seeks to address this issue with a strategy that makes users appear to be insects. With its numerous tiny lenses, the light-field camera technology utilised in the Flamera prototype headset is not brand-new, but Meta has modified it for its needs. One aperture was placed behind each lens in the lens grid by the researchers. These only permit light rays that would normally reach the eye to enter an image sensor. the headphone.
Meta’s idea of distortion-free pass-through augmented reality
This method is referred to in Meta as “light field passthrough.” Even with this strategy, the sensor must first collect the image before processing it to fit the display and lens that the user will see. However, Meta claims that because the curved lens array on the front of the headset captures the outside environment, the perspective of the display is more accurate.
The light that passes through two nearby lenses is frequently combined via technology. Consequently, the shutters make sure that only the light that is necessary for the direction of vision enters. You can see how the other beams are obscured in a caravan.
Conventional light-field cameras, in contrast, catch an excessive number of light beams, which leads to an excessively low image resolution. According to Meta, the new design ensures that the light field’s pertinent areas precisely hit the small number of sensor pixels.
Looking through a colander, the raw sensor data resembles tiny circles of light that only partially reveal the desired image of the outside world. Using a depth map, Flamera rearranges the pixels and computes a reconstruction of the complete image. The end result, in Meta’s opinion, is significantly less susceptible to distortion than current headsets, creating new opportunities for augmented reality applications.
According to researcher Grace Kuo, “by designing our headset from scratch rather than modifying an existing design, we ended up with a camera that looks quite unique but can enable better passthrough image quality and lower latency.” “Flat camera” is the meaning of the name Flamera.
as thin as is practical
Getting thick items, like the circuitry on the camera sensor, out of the way on the road to the current prototype wasn’t so simple. The Flamera optical design functions best with a tiny headset, which allows us to position the passthrough cameras as close as possible to the user’s eyes, according to Kuo.
The new architecture also has the benefit of requiring considerably less computational resources. An complete workstation was required in the previous Neural Passthrough project in order to correct for AI computation artefacts. The camera hardware and reprojection algorithms will cooperate more efficiently from the outset with the new Flamera architecture.
Director of Display Systems Research Douglas Lanman acknowledges that the prototype’s design is still somewhat bizarre. But it already demonstrates one potential future application for perspective-correct pass-through augmented reality. From August 6 to 10, you may view the hardware at the Siggraph conference to see what it looks like.
