Google Unleashes VLOGGER: Revolutionizing Human Video Generation

VLOGGER proposes a groundbreaking method for generating talking human videos using a single input image, leveraging recent advancements in generative diffusion models.

VLOGGER comprises a two-stage pipeline: a stochastic human-to-3D-motion diffusion model and a novel diffusion-based architecture enhancing text-to-image models with temporal and spatial controls.

This approach facilitates high-quality video generation of variable lengths, controllable via high-level representations of human faces and bodies, without individual training requirements or face detection and cropping.

Evaluation across three benchmarks demonstrates VLOGGER's superiority in image quality, identity preservation, and temporal consistency compared to state-of-the-art methods.

A new and extensive dataset named MENTOR, one order of magnitude larger than predecessors, serves as the basis for training and ablating VLOGGER's technical contributions.

VLOGGER employs a two-stage pipeline to transform speech into photorealistic videos, incorporating body motion controls generated from audio waveforms.

The model generates diverse videos while maintaining realism, evident from pixel diversity in generated videos, ensuring varied motion and realistic outcomes.

VLOGGER's applications range from video editing, where it alters expressions, to generating moving and talking people from single input images and driving audio.

VLOGGER edits existing videos, altering subjects' expressions by, for instance, modifying mouth or eye movements, ensuring consistency with original footage.

Several examples demonstrate VLOGGER's capability to generate realistic videos of talking faces from single input images and driving audio.

A major application involves translating videos from one language to another by editing lip and face areas to match new audio inputs.

VLOGGER stands as a groundbreaking innovation in human video generation, promising versatile applications and unparalleled realism in synthesized videos.

MENAFN02042024000228011069ID1108051252