Article on Fiat Lux, Oct 30, 1999
The following article is Copyright © 1999 NewScientist Magazine, reproduced with permission.
|
Armed only with a few mirrored spheres and a digital camera, Paul Debevec set out to make a film like no other. Catherine Zandonella reports SIGNOR Alfredo Pergolizzi brushed a speck of dust from the lapel of his tailored Italian suit and clasped his hands in front of him. Leaning across the desk in his Vatican office, he smiled expansively at the young film maker and said, "Signor Debevec, I am pleased to inform you that we've decided to allow you one hour to take your pictures in St Peter's basilica." Inwardly, Paul Debevec groaned. He had hoped to spend three days capturing the majestic structure in digital pictures for his computer graphics research at the University of California, Berkeley. But after hours of negotiating, this was Signor Pergolizzi's final offer, so Debevec could only smile and accept. And start planning how best to use the time. Debevec, along with fellow researcher Tim Hawkins, had gone to Rome to use the Basilica as a set for an experimental film called Fiat Lux -- Latin for "let there be light". The final product would be a powerful two and a half minute animated film set in the famous building depicting a symbolic struggle between science and the church. Debevec planned to show silver balls crashing chaotically onto the basilica's marble floors and onyx dominoes toppling in thunderous succession.
If he could pull it off, Fiat Lux would be more than just visually impressive. It would be a showpiece for the revolutionary computer modelling and lighting techniques that he and his colleagues had developed over the past three years. The techniques require only a few digital snapshots of a scene to construct a virtual film set in which a virtual camera can shoot from any angle, zoom in and out, and pan and track around the scene. The characters, in this case silver balls and dominoes, are shot or animated separately and added later, but appear as if lit from within the scene. "With Debevec's techniques, virtual models of real-life places could seem more real than ever," says Rob Shakespeare, a professor of theatre lighting design at Indiana University in Bloomington. Since almost any site on Earth can be captured quickly and cheaply in three dimensions this way, the technique has the potential to revolutionise the way films are made. What's more, it can be used to conjure up models for other uses -- to help preserve historical sites, for example, or to make architectural models of buildings. And with easy-to-create virtual film sets comes the potential for a whole new form of three-dimensional storytelling.
Setting a high-quality film in any public space is fraught with difficulties. Filming can take days or even months, but getting permission to use the site for even a few minutes can be difficult, as Debevec discovered. The alternative, at least for well-funded Hollywood directors, is to build a film set in which the actors and actresses perform. But again, this is a time-consuming and expensive business. Another option is to build a virtual model of the scene on computer, film the action separately and add it later. But making a virtual model look real has proved almost impossible -- which is why Hollywood has generally used the technique only for films set in fictional places.
There are two problems that must be overcome to make a virtual film set look real. First, the computer modeller has to create a three-dimensional model that is identical to the original in every way. The shape of every cornice, the texture of every surface and the details in every carving all have to be captured accurately. If this sounds tough, it is child's play compared to the other problem: reproducing lighting accurately. In an indoor scene, this is made up of direct light from sources such as the Sun shining through windows, light bulbs and indirect light reflected from every surface. Lighting is the key to how we perceive reality, and thus whether something looks real, says Donald Greenberg, a pioneer in computer lighting simulations at Cornell University.
One way to make a model is to build it from scratch using a computer software package that provides building blocks in the form of boxes, spheres and other 3D shapes. The blocks are assembled like virtual Lego into a shape resembling the structure being copied. This technique, known as "geometry-based modelling", requires the painstaking efforts of a skilled graphic artist to transform the building blocks into a realistic model.
Instead, a growing number of researchers are turning to image-based methods to create models. Rather than starting with geometric blocks, they start with digitised photographs. Part of the breakthrough that Debevec and his colleagues have achieved is to make this process much easier and the results more lifelike.
Debevec begins with a single photograph of a scene which he feeds into a computer program called Façade. With a mouse or electronic pen, he marks out the boundaries of the walls, roofs and other large-scale structures in the picture. The result is a crude outline of the buildings in the scene. He then selects 3D shapes such as cubes, prisms and curved shapes that roughly match the shape of these structures and asks the computer to orient and size them until they fit the outlines in the scene. For example, Debevec might choose a prism to model the structure of a roof. The computer then sizes and orients the prism until it fits the perspective shown on the screen. At the end of this process, the computer has a 3D skeleton of the structures in the scene. Already, it is possible to fly around this virtual world to see how buildings stand in relation to one another and even to look at the skeletons from behind.
The next stage is to add structures such as doors, windows and even friezes and cornices. Debevec does this using another program that compares the 3D skeleton with the original photograph, and picks out extra details.
The final stage is to "paint" the skeletons to make them appear solid. Debevec uses a method akin to projecting photographic images with a slide projector onto the sides of the model. For this he uses several photographs taken from different angles so that he can project details that were hidden in the original shot, such as the view of the back of a building. Now a viewer can fly around the model and see a realistic view from almost any angle.
Debevec and his colleagues demonstrated these techniques a couple of years ago using the famous clock tower at the University of California, Berkeley, as a subject. In the film, the camera circles around and over the clock tower as if flying in a helicopter. In the background are other campus buildings which all have a 3D structure. Even private houses near the campus appear in the model.
Debevec's technique solves the problem of trying to model a complex geometry and then make it look real, says Michael Cohen, a computer scientist who develops image-based modelling techniques at Microsoft Research in Seattle, Washington. "Debevec builds up models with blocks and then paints them with photographs."
The technique works well for creating a 3D snapshot of a scene, but a virtual film set is more complex. A director will want to insert objects into the scene in a way that makes them look realistic. Light must reflect off them or pass through them in a realistic way and their shadows must fall as if they were really there. The director might also want to add special lighting effects to create a particular atmosphere.
The actual path of light is easy to model -- a simple ray-tracing program does the trick, showing which objects are directly illuminated and which fall in shadow. The difficult bit is understanding how the light reflects off objects in the scene and provides indirect lighting for other objects. Indirect lighting is subtle but extremely important in making a scene look realistic. Without it, colours look flat and the scene seems sanitised and, well, computer-generated.
The first stage in modelling indirect lighting is to measure it. The idea is to measure the light field at a single point in space the light passing through that point from all directions. This is easier than it sounds and Debevec uses a well-known method in which he places a perfect mirrored ball the size of a grapefruit a few centimetres in front of the camera and photographs it. Since the ball reflects light coming from all directions (except directly behind it) into the camera, Debevec gets a pretty good idea of the light field at that point.
Of course, it is impossible to capture the entire range of luminance in a single photograph, as anyone knows who has ever photographed an unlit person standing in front of a bright window. In any scene, whether indoors or outdoors, the amount of light can vary by many orders of magnitude. "In St Peter's, if you compare the underside of the dark bronze of Bernini's baldachino [the canopy over the altar] to the light coming from the windows and from the spotlights in the vaulting, the ratio is well over a million to one," says Debevec. "This is not atypical of rooms with bare light sources and actually less range than one would get outside, where the ratio between looking directly at the Sun and looking into a dark crevice in the bark of a shaded tree is close to 100 million."
To capture this range, Debevec takes a series of digital photographs of the mirrored sphere and varies the exposure in each, first capturing the shadow detail, then the midtones and finally the highlights. Unfortunately, digital cameras do not generate images in which the pixel brightness is proportional to the light in that part of the scene (this is to make their images look more like the ones people expect from photographic film). So Debevec has had to develop software that combines the series of images in a way that takes this into account. The result is a complete record of the light field at a single point across the entire range of brightness levels. With this measurement, Debevec can work out the light field at any other point in the scene, providing it isn't hidden from the camera.
This is highly significant. With the ability to work out the light field at every point, Debevec can insert an object anywhere in the scene and know how the indirect illumination would make it appear from any angle. This is the key technique that allows a director to insert props and actors into a scene.
After perfecting these techniques, Debevec's only remaining task was to create a showcase for them. In Fiat Lux, he wanted to make a movie that was both visually beautiful and displayed the capabilities of these techniques. He chose St Peter's because he knew the complicated interior geometry and different types of light emanating from stained glass windows and overhead lamps would test his techniques to the limit. And the film's title not only explains the goal of the film but is also the motto of the University of California.
The morning after meeting Signor Pergolizzi, Debevec and Hawkins lugged their cameras, tripods and mirrored balls into the basilica. The ancient church was begun in about AD 326 and rebuilt by Renaissance architects in the 16th century. To capture its complex geometry, the two researchers would need to take pictures from several viewpoints so that all angles were covered. Debevec and Hawkins had laid out their plan of action the previous night, using a map to decide on camera locations. Now, on this cool February morning, they worked quickly, taking all the shots they needed within the allotted time. The only other people in the basilica were three nuns and a janitor. No matter, thought Debevec. These bystanders could be eliminated later with a graphics program.
Back in Berkeley a few weeks later, Debevec enlisted the help of a team of undergraduates to work on the project. Using the new techniques, they gradually pieced together the model of St Peter's using just 10 photographs (see Diagram).
Next came the task of adding the synthetic objects -- the mirrored spheres and solemn black dominoes. To make the action look real, the surface of each sphere and domino had to display reflections from the salmon-coloured marble walls and streaming light that filters down from the stained glass windows. As the objects move, these reflections had to move as well.
Debevec also used techniques to give Fiat Lux a cinematic quality. He used slow motion, added soft lighting, varied the camera angles, panned and moved around the room and used a soundtrack of Gregorian chants and crackling thunder.
Because of this attention to detail, it took a hundred 167-megahertz computers and 20 Pentium IIs working full-time for three days to render all the 4500 frames in the animation. When complete, Debevec had succeeded in creating exactly what he set out to make: a concise, authentic advertisement for the growing power of image-based techniques to build highly realistic models.
These techniques promise a new level of realism for Hollywood film makers -- but there are challenges ahead. Debevec can skilfully mimic the play of light on artificial objects such as spheres and dominoes because it is easy to measure and model the way light reflects off their surfaces. However, it is far more difficult to model the way light interacts with human skin or clothing. "Right now there is no interaction in terms of lighting between actors and the synthetic environments we put them in," says Shakespeare.
Debevec is working on a way to capture the way light interacts with the human form. There is no one-size-fits-all approach. The lighting depends on their shape, how fat or thin they are, how they move, what colours and even what fabrics they are wearing. Debevec plans to film actors in a studio under a sequence of different lighting conditions to determine how each reflects light. With this information it should be easier to add them to the virtual film set later.
But film directors will not be the only people interested in Debevec's new techniques. Highly realistic virtual sets can also be used to preserve sensitive environments in virtual archives so that scientists can see how they are changing and so that people can enjoy them in the future when the real thing may no longer exist. Two weeks after the Los Angeles premiere of Fiat Lux in August at the annual computer graphics jamboree known as Siggraph 99, Debevec was on a plane to Egypt as a consultant on a project that will make computer models of the pyramids and other antiquities.
With their similarity to the real thing, these virtual landscapes may be the first step towards of a whole new entertainment experience. One could travel the world without leaving one's living room, says Debevec. And while it may never be possible to reproduce all the delights of foreign travel, there will be some advantages. After all, if one travels to the Vatican on a virtual trip, it won't be necessary to get permission from Signor Pergolizzi to take pictures.
Catherine Zandonella is a technology journalist
based in Santa Cruz
Further reading: find out more about
Paul Debevec's film Fiat Lux at
http://www.debevec.org/FiatLux
|
The four page article in its original layout, without images, is also available in Adobe Acrobat PDF format.
Click here for the main Fiat Lux web site.