With all the similarities between photos and video, how difficult would it really be to transfer this process of panorama making?
We know that to get rid of parallax we need to rotate the camera around it’s entrance pupil, but because we need to capture multiple views at the same and they can’t all be rotated around the same point in space, parallax is going to be unavoidable. How big the parallax errors will be however, is something we can try to control.
The formula for parallax is complex, and is used for everything from observations of space, to hunting rifles, and of course photography. For our purposes, suffice it so say that as the distance from the no-parralax point increases, so does the error.
An extreme example is shown below, with two cameras on either side of a train station. The images are wildly different in their foregrounds, with almost every object being in a different place compared to the background. But as the distance increases, objects further away start appearing in the same place in both pictures. Things like the relationship between the sun, and the clouds are identical between both shots.
For camera rigs with multiple cameras, the goal therefore is no longer parallax removal as it is parallax management. The design of a VR camera array aim to keep all the cameras as close together as is physically possible. For this reason alone, small cameras such as the GoPro Hero series are very popular amongst VR-cinematographers. As will be discussed later though, their size benefits come at a cost.
The number of cameras used also has a ripple effect in the likelihood that the parallax errors will detract from the experience. Fewer cameras may well mean a greater distance between them, and therefore larger stitching errors, but there will be far fewer of them. If shot using only 2 cameras, there would only be 2 areas with errors. More cameras will mean that the individual errors might be smaller, but the shot will be littered with seams.
More cameras also mean more data, more storage and a lot more management whilst introducing many more points of potential failure. Cataloging and storage of footage from even a single camera shoot for a film can be taxing, and with VR, this task multiplies with each additional camera.
The challenge of time
Although Parallax is a big challenge for panoramic capture, it is only part of the story. Video introduces another variable that photographers rarely contend with, continuous motion. Where photography only captures a single point in time, differences in time between the photos in a panorama may very well become invisible. By freezing the motion, photographers are freed from the need for precise temporal consistency – as there is often no way to accurately judge whether the photos were taken at the same time.
Video has a fixed tempo, and that tempo is somewhere between 24 and 30 frames per second depending where you are in the world and whilst the jury is still out on how fast VR needs to be, the fact is that it will be faster than zero.
Videos captured out of sync, when stitched together, exaggerates the effect of temporal inconsistency. The human visual system is incredibly sensitive to movement, and even small imperfections or inconsistencies between two sequences are easy to spot. When an actor walks from one camera’s view into another and they appear a few frames too early or late, the whole experience can be ruined. Even more so if there is any camera movement as different parts of the scene can end up moving at different speeds or different directions.
For a “perfect” panoramic video it is therefore very important that the cameras capture the same moment in time. Though this, as with many of the other challenges of VR can be seen as a chance to experiment. There may be situations where large inconsistencies may in fact be warranted, or interesting and may very well be used in service of a larger goal. But as the old saying goes “know what the rules are before you break them”.
As with any new technology – VR is growing in leaps and bounds and many of the problems mentioned here could very well be solved, as new algorithms are made available to be public. Possible solutions are being developed, but the technologies associated with them are currently in their infancy or worse – proprietary. The current state of the art is discussed to some degree in the ADVANCED chapter.
Although not fundamentally important for creating a panoramic film, the choices regarding sensor size, dynamic range (bit-depth) and compression have a ripple effect on the production pipeline. Some will inhibit the level to which post-production may be carried out or the resolution of the final product, others may simply be a joy to handle, but none are by any means a deal breaker for creating content.
In the end, if synchronization and parallax are addressed, all other considerations for camera selection become largely inconsequential for the ability to create a panoramic film. With enough effort and planning, virtually any camera can be used to create a panoramic film. As such, choosing which camera or cameras to use can generally be addressed to satisfy the needs of the project, or as is more often the case – the budget. For details on the pros and cons of different camera combinations see the next chapter.