Stereographic (3D) Compositing

How do we perceive depth? In theory it sounds easy, we have two eyes separated a certain distance that take two slightly different views on the world. But, if we think about it there are other depth queues that painters know about, and there are other depth queues that are built in to the geometry of a viewpoint. What happens when we want to fool the eye in seeing three dimensions in a VR headset, and how much can we manipulate in post?

Imagine we are shooting an actor on greenscreen with a stereo camera. At a later date we want to move this actor in a virtual environment. How much flexbility do we actually have? We have limited control to some of the depth queues:

1. Stereo separation. We can change the separation of the two images. This changes the convergence point only.
2. Relative size. This is a tricky one. Objects closer to the camera are larger, and this effect is disproportionate as objects get closer. We can't go back later and change this, so there is a small amount of depth we can realistically change after the fact.
3. Angle of view. If we are above an object we will see more of the top than if we are at eye level with the object. If an object is shot close where we see a lot of the top, as we move it away we should see less of the top, but of course we don't.

So is there really any way to change stereo depth after the fact? It appears not significantly. Shots need to be carefully previsualized ahead of time so the green screen actor can be set at the appropriate location in 3D space to get proper separation, correct relative size (no hands grossly enlarged by being too close to camera) and correct angle of view.

Thinking ahead what might be a solution to explore? How about a plane of cameras, a so called lightfield camera? Stay tuned!