What is the Purpose of Eye Tracking in VR?

If you’ve been hearing a lot about eye tracking in VR lately but aren’t sure what it’s actually for, you’re not alone. Eye tracking is one of the most complex parts of VR and mixed reality technology. Only recently has it become truly functional. 

The Apple Vision Pro made headlines in June 2023 for its groundbreaking eye tracking technology. How does eye tracking work, though? What is it supposed to do? 

What is Eye Tracking?

Eye tracking in VR helps you control games and navigate through menus effectively. In mixed reality headsets, it plays a big role in blending your real and virtual worlds. A collection of interior cameras and sensors monitor the movement of your eyes to determine what you are looking at in VR games and apps. 

In the early days of VR, eye tracking was difficult to get right. It wasn’t used much in games and apps because the technology was simply not precise or consistent enough to be reliable. Instead, many early VR games and headsets relied more on head tracking. 

For example, back in 2016, the Google Cardboard project created an explosion in smartphone-based VR. Smartphones lack the cameras and sensors for eye tracking, but they do have gyroscopes and accelerometers. Smartphone VR apps used these sensors to track broad head movements. The player would physically move their head to look at objects on-screen. 

Since head tracking is not very precise, most VR headsets have relied primarily on physical handheld controllers. That all changed in June 2022 when Apple’s Vision Pro headset was announced. This mixed reality headset has no physical controllers. The eye tracking is so precise users are describing it as “magic”, a big step from old-school head tracking! 

VR technology has come a long way over the past several years. Today’s sensors, cameras and processes are powerful enough to consistently, precisely track eye movements. VR users no longer need to move their entire head to select an icon on-screen. Modern eye tracking can reduce VR motion sickness, improve graphics processing and provide a more intuitive user interface. 

How Eye Tracking in VR Works

How do headsets process eye tracking in VR? There are usually a few core components of a VR eye tracking system: cameras, sensors and an AI algorithm. The sensors, such as infrared lasers, measure reflections from each pupil while cameras record eye movements. An AI does the hard work of processing this data and determining where the user is looking. 

Part of what makes eye tracking so complex is VR itself. Since humans have two eyes, each individual eye isn’t necessarily pointing at what someone is actually looking at. The two eyes angle to focus on one point, creating a triangle shape between the two eyes and where someone is looking. 

In VR, the physics of how the eyes work changes significantly. The user’s eyes aren’t actually looking at something in the distance like in real life. They are always looking at something mere centimeters from their eyes, meaning the VR headset has to create the illusion of depth. That triangle shape between the two eyes and the object someone is looking at is essentially cut off in the middle. 

So, the eye tracking AI has to use some advanced math to determine what on-screen object someone is trying to focus on. It artificially creates the depth that you would experience in the real world. Using computer vision and machine learning, the AI is carefully trained to understand the mechanics of eye movement in VR. Eventually, the AI learns to connect slight eye movements with virtual depth in VR apps and games. 

Effective eye tracking doesn’t just help with navigation. It can also improve headset performance using foveated rendering, which renders the exact spot a user is looking at in the highest resolution. This optimized rendering allows games and apps to run resource-intensive graphics more efficiently, consuming less power. 

Eye Tracking in VR: Apple vs Meta

If you’re looking for a way to try out eye tracking in VR, there aren’t many options on the market yet. Good eye tracking is still hard to come by since this technology is extremely difficult to perfect. There are two main VR headsets available that are truly capable of it: the Meta Quest Pro and the forthcoming Apple Vision Pro, which won’t be released until 2024. 

Both of these headsets are pricey, with the Quest Pro starting at $1000 and the Vision Pro at a whopping $3500. So, a VR headset with eye tracking is a big investment. There are a couple key things to consider before buying either of these headsets. 

Eye Tracking Technology

How does the eye tracking technology in Meta and Apple’s headsets differ? The Apple Vision Pro has not yet been released and Apple hasn’t disclosed the exact tech powering their eye tracking. However, the performance of the Apple Vision Pro’s user interface speaks for itself. 

The Apple Vision Pro doesn’t come with handheld controllers like most VR headsets. The UI relies entirely on eye tracking and simple hand gestures. Early reviews and demonstrations show the incredible accuracy of Apple’s eye tracking in VR, with many people even calling it “magic”. 

This may not come as a surprise considering the Apple Vision Pro has a second processing chip exclusively for handling camera and sensor data. Apple also designed the UI from the ground up to primarily use eye tracking for navigation with no need for controller input. 

In contrast, the Meta Quest Pro is currently Meta’s only headset with eye tracking in VR, although it is hardly used. The UI relies on handheld controllers, as do virtually all games and apps playable on the headset. 

However, Meta’s eye tracking technology itself may be comparable to the Apple Vision Pro. One DIY programmer was even able to replicate the Apple Vision Pro’s eye tracking based UI on a Meta Quest Pro. So, the hardware capabilities may be there, Meta simply isn’t making use of them on the same level as Apple. 

Eye Tracking Data Privacy

One very important difference between Apple and Meta is how they treat users’ eye tracking data. Privacy should be a serious concern for users when it comes to eye tracking in VR. This data can reveal what draws your attention, how you focus on things and how you look at digital media. 

All of that information is very valuable to advertisers. So, it’s important to be aware of how your eye tracking data is used in VR apps and games to make sure your privacy is respected. Otherwise, VR advertisers might abuse eye tracking data to put ads in the most inconvenient places in your field of view, ruining the VR experience. 

Apple and Meta have so far displaced opposite approaches to eye tracking data privacy. Apple reassured users from the first day the Vision Pro was announced that eye tracking would remain completely private. It is processed on the device level, meaning eye tracking data never gets ported out to the cloud or some data center. 

Apps and games can use eye tracking data on the Vision Pro, but they don’t have access to the data itself, only the results of it. For instance, the app will only know which icon you’re looking at, not the eye movements that led you to look at that icon. Even Apple itself does not have access to users’ eye tracking data. 

In contrast, Meta’s privacy policy allows them to harvest eye tracking data for a “personalized” experience. This means they can use eye tracking data to sell targeted ads to users just like on Facebook. In other words, not only is Meta accessing users’ eye tracking data, they are likely planning to give advertisers and other third parties access to it, as well. 

Eye Tracking: The Future of VR Navigation

Eye tracking in VR can make virtual experiences seem like magic or mind control, making a truly intuitive user experience. If eye tracking data is handled responsibly, tech companies like Apple can use eye tracking to transform mixed reality. The next generation of VR will likely use a combination of input methods, including intuitive eye tracking alongside controllers and keyboards.