MotionLLaMA: MotionLLaMA: A Unified Framework for Motion Synthesis and Comprehension

Zeyu Ling*

Zhejiang University, Zhejiang Lab

Bo Han*

Zhejiang University

Shiyang Li

Zhejiang University

Hongdeng Shen

University of the Chinese Academy, Zhejiang Lab

Jikang Cheng

University of the Chinese Academy, Zhejiang Lab

Changqing Zou

Zhejiang University, Zhejiang Lab

Paper Code

This paper introduces MotionLLaMA, a unified framework for motion synthesis and comprehension, along with a novel full-body motion tokenizer called the HoMi Tokenizer. MotionLLaMA is developed based on three core principles. First, it establishes a powerful unified representation space through the HoMi Tokenizer. Using a single codebook, the HoMi Tokenizer in MotionLLaMA achieves reconstruction accuracy comparable to residual vector quantization tokenizers utilizing six codebooks, outperforming all existing single-codebook tokenizers. Second, MotionLLaMA integrates a large language model to tackle various motion-related tasks. This integration bridges various modalities, facilitating both comprehensive and intricate motion synthesis and comprehension. Third, MotionLLaMA introduces the MotionHub dataset, currently the most extensive multimodal, multitask motion dataset, which enables fine-tuning of large language models. Extensive experimental results demonstrate that MotionLLaMA not only covers the widest range of motion-related tasks but also achieves state-of-the-art (SOTA) performance in motion completion, interaction dual-person text-to-motion, and all comprehension tasks while reaching performance comparable to SOTA in the remaining tasks.