Motion Capture Records Dots on a Suit and Builds the Performance Later. Here Is the Pipeline, Stage by Stage.

A performer walks into a room full of cameras wearing a suit covered in dots, plays a scene, and walks out again. The character the audience meets months later doesn't exist yet on that day. Motion capture records the dots, and the performance on screen gets built afterwards, in stages.

The stages run in a fixed order: the suit and its markers, the volume the cameras watch, capture day, cleanup of the raw data, retargeting onto a different body, a separate pipeline for the face, and the render. Only the render shows anybody a character.

The screen performance gets built later, starting with dots on a suit

The suit carries the markers, and the markers carry the whole pipeline. On Dawn of the Planet of the Apes, Wētā Digital put stronger LEDs inside flexible silicon on the actors' suits, and fxguide reports why: the markers stayed readable in dark conditions, in forests and on night shoots, and the performers kept a full range of motion. Together, the markers describe a body in motion, joint by joint, and every later stage works from those points and nothing else.

The volume is the space the cameras watch, and the Apes films took the volume outdoors

Every capture happens inside a volume: a space ringed by cameras, calibrated so each marker shows up for several cameras. House of Moves, the Los Angeles studio fxguide profiled, shot one commercial on a smaller stage holding 80 Vicon T160 cameras around a volume of 70 feet by 40 feet. The larger stage ran 200 cameras. Scale buys coverage: fewer blind spots when performers cross, roll or fall.

The Apes films shrank the rig and carried the rig outside. Rise of the Planet of the Apes broke the studio habit and extended the volume into exterior locations, including the Golden Gate Bridge sequence. For Dawn, Wētā Digital deployed portable volumes built to survive sleet, rain, extreme heat and walls of flame 50 feet high, working in forests, at a dam and on New Orleans locations. A complex set could call for up to 40 motion capture cameras, fxguide reports.

Capture day records markers and reference video, and the character stays absent

On the day, the performers play the scene for real. The cameras record marker positions, and reference cameras record the same performance in ordinary pictures. On Dawn, Sony F3 cameras handled reference duties in the full setup, and each actor wore a single head-mounted face camera recording high-definition pictures at a high frame rate. Every stream ran in sync, matched to camera shutter times.

The take leaves the volume as data: a moving cloud of points, plus video reference the team can check the points against. Nobody has seen the character yet. When time or access ruled out the full volume on Dawn, the crew dropped to a reduced setup called faux-capture: reference cameras and a triangulated point-cloud solution, with less data and more repair work downstream.

Cleanup crews repair the raw data before anybody animates a frame

Raw capture arrives with gaps, swaps and noise. A marker hides behind a body, two markers trade identities, a point jitters. House of Moves cleaned its capture first in the studio's own proprietary software, fxguide reports. At Wētā, a tracking team cleaned the data and made sure the performers looked right, in motion capture supervisor Dejan Momcilovic's account to fxguide. Cleanup sounds like housekeeping. Cleanup decides how much of the original performance survives.

Quantic Dream prices dirty data in plain terms. The studio shot motion capture three to four days a week for a year and a half on the road to Detroit: Become Human. Animation director Jan Erik Sjovall told fxguide the studio once outsourced its cleaning, solving and retargeting, then brought the work in-house after inconsistent, spotty results. His reasoning: when data comes in at low quality, animators fix the data until the shot works, and the fixed shot no longer matches the performance the actor gave. Sjovall states the animator's job the other way round: animation enhances a performance and never replaces one. His own figure, labelled as his: 80 to 90 percent of the game's animation came from motion capture.

Retargeting moves the cleaned motion from the performer's body onto the character's body

Clean data still describes the wrong body. The performer stands human height, with human hips and arms. The character might stand as an ape, longer in the arm and shorter in the leg. Retargeting translates the motion from one skeleton to the other. On the Apes films, Giant Studios tracked, solved and retargeted the capture data before the data entered MotionBuilder, and Wētā's motion editing department then ran a further retargeting pass — Momcilovic calls the human-to-ape system pretty elaborate — and finished by fixing the character's contact with the world: hands on props, feet on ground.

Joe Letteri, Wētā's senior visual effects supervisor, explains to fxguide why a copy can't work. Even with arm extensions, the captured body never matches ape limb lengths and joints exactly. The hips land wrong, or too high, and moving the hips down changes the head and the gaze. Every fix moves something else, so retargeting answers with judgment, shot by shot, protecting the acting choice the performer made.

The face runs through a solver that turns marker positions into muscle movements

Faces get a pipeline inside the pipeline. A face camera gives the team a sparse set of moving points, and the points have to drive a complex digital face. On Dawn, Wētā's face solver took the head-rig data first, produced blend shapes and facial muscle movements, stabilised the result, and translated the performance from a human face onto an ape face. Animators then adjusted lip sync and corrected expressions on the new face. Certain shape combinations can push a face off-model, into an expression the character can't actually make, and the system fires a corrective shape to head the error off.

Wētā's published account of Gemini Man shows the same stage built for a human character. The team ran Will Smith through facial exercises, raising his upper lip and working through movements broken into action units, the elements of the Facial Action Coding System. Smith wore facial markers and a head-mounted camera rig for the recordings, and Wētā calls the facial side performance capture, set apart from motion capture of the body, because the facial data arrives at extremely high quality. The team's solver represented the captured marker positions as action units, trained with deep learning so the breakdown matched how its animators think, and the solver could then decide, frame by frame, which muscles were firing. Stuart Adcock, Wētā's facial animation supervisor, compares the result to a musical score: a score written from one instrument can get played on another, and the second instrument keeps its own sound.

The render gives the performance skin, fur and light months after the shoot

The last stage assembles everything the earlier stages saved. The retargeted body and the solved face land on the finished character, inside a lit set, and the renderer computes the picture. Most of Dawn was rendered in RenderMan, fxguide reports, and Wētā's newer path-tracing renderer, Manuka, handled a limited run of shots, including the show-of-force sequence in San Francisco, where crowds of fur-covered apes filled the frame. The film's Deep compositing pipeline ran as standard on every computer-generated shot, inside a Nuke workflow, so compositors could adjust characters late, even inside smoke, fire and dust.

Finish work runs long. One signature Dawn shot, a colony attack built around a spinning tank, ran past 1,000 frames and took 740 render passes, with four compositors working in Nuke for over three months. House of Moves shows the smaller end: frames rendered in the Unreal game engine at roughly three to four minutes a frame, fast enough to re-render changes the same day. The shape holds at both scales. The suit records dots. People turn dots into a performance. The render turns the performance into a picture, and the picture finally looks like the thing the audience paid to see: an actor, acting. Wētā once handed the whole chain to fans at New York Comic-Con, fitting them with partial capture suits, seating them on a mechanical horse simulator and mapping them live onto the digital Caesar. The fans supplied the dots. The pipeline did the rest.