“Creating 2D Animation” by Paolini

Digital Compositing and 2D Animation

Compositing and 2D animation is an exciting field for digital artists, effects editors, and designers. There are many opportunities in both the broadcast video and feature film markets for visual effects artists. Today more than ever, there is an increasing need for people with that special blend of creative and technical skills. Not only does one need to understand the basic “tools of the trade,” which can include a variety of software, one must also be aware of the visual nature of the process.

Remember, the bottom line is that all the technical considerations are unimportant when confronted with the question: “Does it look right?” Obviously this is a subjective judgment, and a good compositor able to make these decisions will always be in high demand.

What is a Composite?

Compositing is simultaneous multi-layering and design for moving pictures. It is the digital blending of a background plate and one or more foreground elements to create one seamless, well-integrated film image. Modern designs often use many techniques
together, such as painting, retouching, rotoscoping, keying/matting, digital effects, and color correction as well as multi-layering to create complex animations and opticals for promotions, title sequences, and commercials as well as program content. Besides the creative element, there are other important applications for compositing such as image repair, matte painting, and wire removal – especially in motion pictures. The artist and the equipment can be crucial, especially where seamless results are demanded.

Digital Compositing Tools

The tools used in the process of creating 2D animation and compositing include:

Paint programs
Color correction utilities
Warping and morphing tools
Matte-extraction software
General-purpose compositing packages

Basic Terms
Elements

Elements are your images. You may also commonly hear elements referred to as “layers.” A subset of elements, called “plates,” usually refers to original scanned sequences. Intermediate elements generated during creation of a composite are not referred to as plates, but as “precomps.”

Mattes
A matte is an image designed to control the transparency and opacity of another image. Mattes are used during compositing when we only wish a portion of one image to be included in the output image. You may also hear the term “mask” used when referring to mattes, and in general the two terms are usually interchangeable. “Mask” is more common when specifying an image that is used to control or limit a color correction (or some other form of image-processing) on another image.

Matte Generation

There are many different types of mattes, and there are many different methods used to generate mattes for compositing. This process, particularly when automated, is also referred to as “matte extraction” or “pulling a matte.”

There are two approaches we can take to generate these mattes. The first is hand-drawing a matte for the object in question over every frame of our sequence. This is still occasionally done, but only after all other options have been exhausted, since the process of hand-drawing a matte for every frame of a sequence is time-consuming and error-prone.

One slightly less brute-force method of generating a traveling matte involves the use of splines to “rotoscope” basic outlined shapes for an object. The rotoscope artist can specify certain key shapes, and the software will smoothly interpolate any in-between frames. Unfortunately, objects that move or change shape a great deal may end up needing a key shape defined for every frame anyway!

There must be a better way, you say, and there is. Instead, we rely more on procedural techniques, where some initial parameters are determined that are capable of extracting a matte, and then software is allowed to apply these same parameters over a sequence of images. Some of these methods include The Color Difference Method, which involves photographing the subject in a manner that greatly simplifies the extraction of a matte.

The Color Difference Method requires shooting the foreground object we wish to isolate in front of a uniformly colored backdrop. Any color for the backdrop (or “backing”) may be used, as long as the foreground is essentially devoid of this color. The
term “color difference” refers to the difference in color between the foreground and the colored backing.

The most common backing colors are blue and green, and the choice of which to use is generally determined on the basis of the subject’s colors. If someone says that the subject must be wearing a blue shirt, then typically a green-screen shoot would be dictated. Generally, tests are done if the choice is not obvious. The process of extracting a matte with this method is known as “keying,” and the extracted matte can be referred to as the “key.” The software used to key something from its background varies in complexity and capabilities.

There is also a method of creating a matte known as “difference matting,” where a frame of the scene without the subject is subtracted from a frame with the subject. In theory, all you are left with is the subject. In practice, slight lighting differences, shadows, and grain make the difference between the two images unpredictable. Difference matting is sometimes the only solution available, and it is actually a very useful first-pass that can then be cleaned-up by hand.

Even the best tools can have problems with certain images, and real-world situations often deliver to the compositor plates that are less than perfect in terms of evenness, graininess, and absence of objects that are not intended to be seen in the final
composite. To help deal with these, one almost always creates garbage-mattes around the subject. These are loose-fitting shapes designed to quickly remove problem areas from the scene.

Several of the techniques mentioned will often be used in conjunction with one another, combined until as flawless a matte as possible results.

Image Tracking and Stabilization

When photographing an element to be used for visual-effects work, one sometimes has the ability to specify that the camera be unmoving, or locked-off, for the duration of the shot. However, it is often not possible, or even desirable, to do this. Multiple shots without camera moves can become boring and lifeless. In situations where the need arises to composite elements that were shot without identical camera moves, one must resort to tracking. Tracking is the process of selecting a particular region of an image and determining that region’s movement over time (on a sequence of images).The data are stored as a series of moves or positions and then applied to one or more images.

There are a variety of situations where tracking can be used. One reason would be to “stabilize” the sequence you are working on. Another reason would be the need to synchronize the movement of an object you are adding to the scene, with something already in the scene. (The object in the scene may be moving, or the camera may just be moving relative to the object).

Preparation of Elements

The best composites are those whose elements were planned and photographed with the explicit intention of creating a composited image. While this may seem like an obvious, elementary statement, you’d be surprised at how often it is ignored in the
real world. In just a bit, we’ll look at what can be done to help fix improperly shot plates, but first, let’s look at some things that can be done to make everyone’s lives easier.

Matched Lighting

Whether you are planning to integrate elements shot with blue- screen or synthetic CG images, or are just soft-splitting two plates together, it is critical that the different pieces look like they were lit with the same lights. Lights should hit the objects in the scene from the same angle, have the same apparent intensity, be of the same color, etc.

Matched Cameras

Almost as important as the lighting is the synchronization of the camera for all elements. Be aware of the camera’s positioning relative to the subject and the height of the camera from the ground plane, and ensure that the same size lens is used when
photographing all plates. If your camera moves throughout the shot, either plan on shooting it with a mechanical motion-control move (a device that allows the camera to repeatedly execute the exact same move) or be prepared to do a lot of post-processing tracking to try and duplicate the move.

Film Stock

If possible, shoot all elements with the same type and speed of film (and expose and develop them similarly). Different film stocks have widely different grain characteristics, and the discrepancy can be obvious.

Element Repair

Fixing Elements

The usual situation in this business is that by the time you receive the elements for your composite, there is no longer an opportunity to re-shoot anything to correct faulty plates. At this point, your only option is to manually fix the problems. The following section lists potential and common pitfalls and gives a few solutions. Remember, every shot has its own problems, and the true test of a good compositor is the ability to come up with efficient and creative solutions to these issues.

Wire Removal

It’s very common for visual effects to be mixed with practical effects that require things like gag-wires, harnesses, ropes, and other mechanical devices. It is not always possible to fully hide these items from the camera, so digital effects may be needed to remove them. There is limited commercially available software for wire removal, and the general consensus is that this software works well on the easier situations, but the more difficult shots are still going to be done at least partially by hand.

Plate Instability

As mentioned earlier, tracking software should be used, either to stabilize the plate or to “bounce” the new elements to match the background.

Mismatched Lighting, Cameras, Action

Trying to tie together images whose lighting doesn’t match can be one of the most frustrating tasks a compositor can undertake. Troublesome highlights on objects can often be decreased via specific color-corrections and masking. When your A and B plates were accidentally shot from wildly different positions or with different length lenses, you may need to compensate by moving and scaling the various elements relative to each other. Even the timing of the action may not initially work correctly. Not only can this be dealt with by “slipping” the synchronization between the plates (so that frame 1 of element A is combined with frame 30 of element B, for example), but you can also adjust the speed of action by dropping, duplicating, or averaging frames.

Hand Painting

Ultimately, for certain problems, there may be no other solution but to hand-paint the offending area, possibly on every frame. This is usually somewhat of a last resort, but the fact remains that it is a totally valid and acceptable solution. Sometimes you may be able to hand-paint a fix on a single frame and then use tracking and warping tools to apply this fix to the rest of the sequence.

The Final Touches

The more time you spend compositing, the more you’ll learn about what things are important in order to fully integrate the elements. There are a number of techniques (and several tricks) that can be used to trigger the visual cues the eye is accustomed to seeing. There are also a lot of common problems that can be easily addressed if they are identified. First and foremost, a good compositor should understand how a real camera behaves. You will often need to mimic artifacts and characteristics of shutter, film, and lens. Many of the items mentioned below are related to this issue.

Blur Levels

It is rare that every element in a scene is in sharp focus. Depth of field dictates that objects farther or closer than the focus point will grow more and more unfocused. Determine the distance your element should be from that focus point and blur accordingly. This focus relationship can change over time (rack focus). Be sure to match the animation timing as closely as possible. Also, keep in mind that a moving object, when recorded on film or video, will have “motion blur. ”This is due to the distance the object moves while the film is exposed (or the video camera is recording). If you wish to place a moving object into a scene, and that movement is something that you created (either as a 3D element or with a 2D transformation), you should plan on motion blurring your element. Most high-end systems allow motion-blur to be added to a 2D move or rendered with the 3D element.

Lens Flares

In the real world, when a bright light source shines directly into a lens, you will get a flare artifact. It is often desirable to duplicate this flare when creating an image that has bright light sources that were not present in the original elements. Be careful with this effect, as it has become over-used in much of the CG imagery being created these days.

Film Grain

Another extremely common mistake with adding pure CG elements to a scene is to ignore the amount of grain that the rest of the plate has. The CGI element will consequently appear far too “clean.” In general it’s much easier to add grain to an element than it is to remove it, so for this reason effects photographers try to use the least-grainy film they can get away with.

Interactive Lighting

If your background scene has flickering or inconstant lighting, (and your foreground plate doesn’t), you will need to do your best to match the background. In some cases it may be as simple as adding a fluctuating brightness effect that is synchronized to the background. In other situations, you may need to have articulated mattes controlling the light so that it only falls on certain areas.

Edges

Examine the quality of the edges of objects that are already in the scene and try to match them as closely as possible. Edges can be sharper or softer depending on the amount of backlight an object is getting or how out-of-focus it is. Be particularly aware of how edges are behaving over time. What looks acceptable on a still frame may “chatter” in the moving sequence.

Shadows

A common novice mistake is to forget the fact that an object should cast a shadow (or several shadows). Many methods of extracting an object from its background do not allow the object’s shadow to be brought along. In this case, you’ll need to create a shadow yourself. It is often acceptable to simply flop the matte of the foreground element and use it to darken a section of the background. Remember to match the rest of the shadows in the scene, in terms of size, sharpness, and density.

Black Levels

Black levels in the elements you add should match the background plate. This is probably the most often-violated compositing rule, particularly when integrating CGI imagery with live-action. Rarely are the darkest parts of a scene pure black, and if you want to add something new to that scene, you should make absolutely certain that the darkest parts of your new element aren’t any darker. An excellent trick for ensuring that your element’s levels are matched to the background is to adjust the monitor  brightness to high and low extremes. This will help to bring areas that the human eye is less sensitive to into a range where differences become more obvious.

Camera Movements

The nature of effects photography often requires that the elements of a composite all be shot with an unmoving, locked-off camera. While this makes for easier composites, it doesn’t really make for interesting (or believable) cinematography. Fortunately, it is possible for the compositor to add camera moves after the plates are shot. This move may be as simple as a nearly imperceptible camera shake, or as complex as a
tracked, 3D match-move.

Atmosphere

Often there may be fog, smoke, or haze in the scene you’re wishing to integrate with. The farther away from the camera your element is supposed to be, the more atmosphere would need to be added. Again, examine other elements in the scene that are at the same distance and try to match their levels. If the atmosphere in a scene is very uniform, you can probably get away with just decreasing the contrast in your element and adjusting its color. With more distinctive mist or smoke, you may need to use a separate smoke element and add it in explicitly.

Rendering

Last, but not least, you have to render your animation. The computer will render each frame in turn, each comprised of various operations to achieve the desired effect. The rendering may take minutes, hours, or days depending on the effect’s length and complexity and, most importantly, the speed of your computer.

Parting Thoughts
Finally, remember that every person who views an image has a little expert he carries around with him. This expert, the subconscious, has spent a lifetime learning what looks “real.” Even if, consciously, the viewer is unable to say why a composite looks wrong, the subconscious can notice warning signs of artificiality. Beware.