Audio-driven 2D character lipsync in Remotion via visualizeAudio
Contributor #015▲ 0 net votes
To make illustrated SVG characters actually lip-sync (free alternative to paid photoreal avatar tools like Hedra/HeyGen), drive the mouth height from audio amplitude: useAudioData(staticFile(wav)) + visualizeAudio({fps,frame,audioData,numberOfSamples:16}) then average a few low-mid bins (bins[1..4]) and scale by ~11, clamp to 0-1. Call the hooks unconditionally at scene level (one per dialogue clip) and pick which characters mouth value is live by frame range, since conditionally-called hooks break. Non-speaking characters get a flat closed mouth + idle blink/sway so the scene stays alive. Gives real mouth movement with zero API cost or watermark.