Watch bad 3D animation and the eyes forgive, the hands forgive - the mouth does not. Lip sync is where credibility lives, which is why every animation pipeline eventually builds one: an audio file in, a viseme timeline out, and a rig mechanism that turns that timeline into motion. The automation is real and it is worth using. It is also, reliably, a first pass.
The two mechanics
| Mechanism | Best for | Watch for |
|---|---|---|
| Shape keys | Stylised characters, VTubers, per-syllable control | Key hygiene - unlabelled keys become a swamp |
| Jaw and mouth bones | Rigged characters, deformable faces, mocap stacking | Rig quality - a bad jaw is a bad scene |
The automatic pass
01 - Prepare the audio. A dry, close voice recording - no music bed, no compression war. Every extra sound source is a syllable the tool will try to lip-read.
02 - Choose your weapon. Open-source analysers that emit viseme timelines, addons that drive shape keys directly from audio, or scripted pipelines that key a rig from a transcript with timing. All three solve the same problem: time + phoneme.
03 - Map visemes to the rig. Whatever the tool outputs - a list of mouth shapes with timings - has to land on your specific mouth. Fat mouth shapes map to wide keys, closed positions snap the jaw shut: a mapping table built once and reused forever.
04 - Accept the tempo, then fix it. Play back with your eyes closed first: if the mouth rhythm matches the speech, the pass worked. Everything after this point is taste, not technique.
The human pass: four fixes that sell it
Co-articulation. Real mouths glide between sounds; generated keys snap. Ease the transitions by a frame or two each way and the machine look evaporates.
Emphasis. Stressed vowels deserve a wider mouth, quiet ones can underplay. Flatten a whole sentence to the same intensity and the performance reads as a puppet.
Breaths and micro-poses. A millimetre of breathing motion between phrases keeps the face alive through silence, which is where machine lip sync is at its most machine.
Eyes and head. The mouth is a third of the performance. Two blinks in the right places and a head that moves with the emphasis will forgive a mediocre viseme pass. Without them, a perfect mouth still reads as animatronic.
The VTuber route
Live avatars do not use recorded audio at all - tracking runs the mouth in real time, and audio-driven lip sync is for recorded content. If your character lives in both worlds, plan for two systems driving one mouth: tracking for streams, audio-driven keys for produced clips, and a rig honest enough to take both. It is a solved design problem; it just has to be decided before the shape keys get named, not after.