BAAI and Peking University Develop Joint Audio-Video Editing via Single Text Instruction
Researchers from the Beijing Academy of Artificial Intelligence (BAAI) and Peking University have developed "InstructAV2AV", a method that performs joint audio-video editing using a single natural language instruction. This research, accepted at SIGGRAPH Asia, allows both visual and audio elements to respond to instructions in a single end-to-end generation process. This approach eliminates the need to edit video and audio separately or manually draw masks, ensuring synchronized multimodal outputs. It significantly advances generative media by enabling more intuitive, cohesive, and efficient content creation. To address the extreme scarcity of paired audio-video editing data, the team built an automated pipeline and released "InsAVE-80K", the first large-scale dataset for instruction-guided joint audio-video editing. The model edits the specified visual objects and their corresponding sounds simultaneously without requiring user-defined bounding boxes.
## BACKGROUND
Traditional video editing tools often treat visual and audio tracks as separate entities, requiring editors to modify them independently and manually align them. While AI-driven image and video editing have advanced rapidly, joint audio-video editing has lagged due to the high cost of creating datasets that contain synchronized, edited video-audio pairs.