The Experience of Virtual Reality: Viewing virtual worlds

The current rise of popularity of virtual reality can be placed squarely upon one factor, the availability of hardware that can delivery on the promises first made in the early nineties (and before). The most important part is to delivery a view of the virtual world that can delivery an immersive experience that is comparable to the real world, even if that world is simplified.

Current VR is delivered using stereographic imagery, that is a slightly different image for each eye that allows us to see the world in 3D. This work like a View-Master toy, showing eye something different. In terms of delivering the visuals to the user using flat displays there are four main factors that need to be covered.

  • Viewing clarity.
  • Frame-rate.
  • Fidelity of the world.

In more detail.

Viewing clarity

The clarity of the a virtual reality display is made from two things, the field of view, that is how much can the eye see in environment, and the pixels per degree, that is how clear any part of the field of view is. With phone display at a normal distance the phone can take around 30 degrees of what someone can see. With a VR display we need that same display to take up as much of our field of view as possible, or else we feel we are looking at the world through a tube or it feels we're wearing googles! If we keep our head still and can move our eyes then each eye has a viewing angle of 120 degrees left and right and up and down. Currently the best VR headset manages 110 degrees, with the Cardboard and Gear VR offering around 90 degrees. This means that each eye sees less pixels so even though a phone display may appear to smooth in normal use, close up you can see the spaces between pixels and each pixel gets bigger. The gap leads to what is called called the screen door effect, which is like having a fine mesh suspended in front of the world. The bigger pixels lead to a blurrier world.

If we look at current phone screen technology we can see that they have various resolutions packed into a certain space. This is known as the pixels per inch. For example an iPhone 6 has a PPI of 326 pixels and the Samsung S6 has a PPI of 577. If we take the PPI of current devices and divide it by viewing angle we get something like this. The higher the pixels by degree the better.

Device ResolutionField of ViewPixels per degree
Google Cardboard 1280x768 to 2560×1440 90 degrees + 7-14
Samsung Gear VR 2560×1440 (using S6/S7) 96 degrees 13
Oculus Rift 2160×1200 110 degrees 10
PlayStation 4 VR 1920x1080 100 degrees 10
HTC Vive 2160x1200 110 degrees 10

From this we can see that the Gear VR appears to give the best clarity with a pixel of degree of up to 13. And this is still noticeably blurry to new users. But it is only part of the story. The screen technology of the displays for the higher end displays provide better pixel updating so avoid the black trail that can be seen on a Gear VR. The wider view angle does help with immersion and the Gear VR and Cardboard can make the view feel like they are wearing a diving mask instead of being directly in the world. Out of these listed only the Playstation VR is looking a bit left behind, we'll say more about this later. It's worth noting that whilst only a few more degrees of viewing angle are needed the pixel per degree has a lot of room for improvement.

The next important factor is how quickly the screen refreshes.

Frame-rate

The amount of updates per second makes a bigger difference in VR than elsewhere. Whilst film can get away with a pedestrian 24 frames per second (each frame being shown twice) VR requires something much closer to reality. Lower rates leads to the user feeling ill after heavy use (a subject for a different post). Sixty frames per second is seen as the entry point for VR. 75 frames and is the current target.

Another way to look at it is to say that the VR experience needs to pump enough pixels per second to each eye to make the experience compelling. Time the resolution by the amount of frames and the numbers start to add up. It easy to see why we need high performing technology.

But there is more to add to the requirements of the display and the machines driving the display.

Fidelity of the world

If we are talking about virtual reality then we are aiming for as much reality as possible. This is where phone based VR and desktop/console based VR depart. At the moment Gear VR experiences are simplistic and kept to simple shapes and objects. This allows the frame rate to be kept high enough.

Gear VR Games

Comparatively Oculus and Vive games are pushing the power of new graphics cards, like the new Nvidia GTX1080, a new bit of PC hardware with VR squarely in it's sights. Gamers expect their game worlds to have the detail and effects comparable to that they seen on a the flat screen - and as the experience is being pushed to both eyes that means at lot of grunt work.
It has meant that Sony is in a position where they realise the Playstation 4 is (or was if you're reading this in the future), not really up for the job. At the time of writing it is a badly kept secret that Playstation will be evolving their console to allow them to pump enough pixels to keep their new VR headset in the game. This is codenamed the Playstation Neo and has been dubbed the playstation 4.5. We should soon see what they deliver.

So we can see that technology has only just got to the level that it can delivery the best visual experience.

The missing 3D cue: Focus

At the start I stated that current VR uses flat screens and stereographic images to give a 3D view. This allows our natural binocular vision to place objects in 3D space. Yet we also use our ability to focus on objects to provide extra information about where an object is in space. For some, this missing focus means that VR always feels a bit flat. So how can we add the ability to objects back into our virtual worlds. Well we could track the eyes focus and change the flat world view whilst adjusting the apparent focus of the flat screen in-front of the viewer. But this is very complex, requires a lot of moving parts and is unlikely to give exactly the same results as real life. The other way is to use a new technology that is currently available only as a capture technology - Light Field.

Light Field technology turns each pixel that is captured into a set of data about where light hitting it came from. The upshot is that the image is not a flat image at a fixed focus but a Light Field where the point of focus can be determined. It also captures enough information to allow slight position shifts as well. The two images below shows to selected focus points from the same Light Field capture. The world leader in this is Lytro who have produced two still cameras that can capture these images.

Lytro are now working on a cinematic camera and a VR capture system. This VR system allows the user a degree of movement within a real world capture of a space. It is called Lytro Immerge

But how can this be played back to create VR? The answer is through ditching flat displays for a display that uses a refraction grid to recreate the Light Field. In effect each pixel becomes a tiny world view that aims it's light in the direction the light originally or virtually hit it. The end result is the image produced can be focused on! The world leader in this technology is MagicLeap, who is using it to create Augmented Reality. The ability to focus on an object means that it can be placed in the real world with other objects at the same focal length.

The bad news is that this means each pixel becomes effectively a small picture. This means the processing power needed goes through the roof. The good news is that refraction grid displays tend to be very light.

Here is a real video of the technology in action showing the change of focus happening on virtual objects.

For more on lightfeild capture see Lytro
For more on lightfeild displays see Magicleap

Next

I'll be looking at moving around in virtual worlds.

Comments