CASIM: Composite Aware Semantic Injection for Text to Motion Generation

Che-Jui Chang*1,2, Qingze Tony Liu*1, Honglu Zhou3, Vladimir Pavlovic1, Mubbasir Kapadia4
1Rutgers University, 2Amazon, 3Salesforce AI Research, 4Roblox
*Indicates Equal Contribution
Paper Code Demo arXiv

TLDR. We propose CASIM, a composite-aware semantic injection mechanism for text to motion generation that exhibits stronger text-motion correspondence and better generalizability.
(Left) Fixed-length semantic injection, which primarily relied on the [CLS] token embedding from CLIP to represent the entire text prompt, fails to capture the subtle differences in individual words. As a result, it generates highly similar motions from distinct text prompts. (Right) Our Composite aware semantic injection method allows each motion frame to dynamically attend to every word token (e.g., “left” or “right” hand), enhancing the motion-text correspondence.

Abstract

Recent advances in generative modeling and tok- enization have driven significant progress in text-to-motion generation, leading to enhanced quality and realism in generated motions. However, effectively leveraging textual information for conditional motion generation remains an open challenge. We observe that current approaches, primarily relying on fixed-length text embeddings (e.g., CLIP) for global semantic injection, struggle to capture the composite nature of human motion, resulting in suboptimal motion quality and controllability. To address this limitation, we propose the Composite Aware Semantic Injection Mechanism (CASIM), comprising a composite aware text encoder and a text-motion aligner that learns the dynamic correspondence between text and motion tokens. Notably, CASIM is model and representation-agnostic, readily integrating with both autoregressive and diffusion-based methods. Experiments on HumanML3D and KIT bench- marks demonstrate that CASIM consistently improves motion quality, text-motion alignment, and retrieval scores across state-of-the-art methods. Qualitative analyses further highlight the superiority of our composite aware approach over fixed-length semantic injection, enabling precise motion control from text prompts and stronger generalization to unseen text inputs.

Architecture

CASIM consists of two major components: Composite Aware Text Encoder (Left) for extracting granular word-level embeddings and Text-Motion Aligner (Middle) for aligning motion embeddings with relevant textual embeddings inside a motion generator. The Text-Motion Aligner can be integrated with three genres of motion generation models (Right).

Quantitative Results on HumanML3D & KIT-ML datasets

Tab 1 & 2
Tab 3 & 4

Qualitative Results

BibTeX


        @article{chang2025casim,
          title={CASIM: Composite Aware Semantic Injection for Text to Motion Generation},
          author={Chang, Che-Jui and Liu, Qingze Tony and Zhou, Honglu and Pavlovic, Vladimir and Kapadia, Mubbasir},
          journal={arXiv preprint arXiv:2502.02063},
          year={2025}
        }