Back

Realtime Avatar Stream

An experiment around realtime 3D avatars, lip sync, Audio2Face and GPU orchestration.

Notes

At the intersection of work and curiosity, I wanted to dig into Nvidia's Audio2Face model, and how to build a live 3D avatar. Audio2Face has been made opensource in 2025 and can be used unlicenced - as long as it is deployed without the NIM Nvidia container. I got to build the orchestration to stream audio from openai/elevenlabs with face blendshapes, learning a lot about 3D, unreal engine, metahuman, and webgl in the process.

After being able to control the face from the audio, I really wanted to give the AI abilities of a "director" - controlling facial emotion precisely in sync with the audio.

This was done at the cost of a small initial latency : rather than tapping directly in a multimodal model like openai livestream api, I first generated text from a fast llm, with inclusion of xml tags describing actions. The streamed text is send to a tts api, and splitted on action tags. With that, we can precisely time the action trigger in the speech stream.

For the need of the cheap demo, I also had to devise some GPU orchestration system to orchestrate cheap runpod instances only when I need them.