Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking

2026-07-07

AI Summary

This work addresses the challenges of semantic mismatch and insufficient temporal alignment in video-to-music recommendation by proposing VTMR, a two-stage framework. It first performs coarse-grained semantic retrieval via global embeddings within a unified audio-visual-text representation space, followed by a fine-grained re-ranking of candidate music tracks using a temporal attention mechanism to model precise time-aware correspondences between video and music. By jointly optimizing semantic compatibility and dynamic temporal alignment, the method achieves strong performance on public benchmarks, attaining an R@10 of 18.3 and reducing the median rank to 46. Human evaluations further demonstrate that the recommended music quality surpasses that of generative baselines and is comparable in overall user preference to leading commercial systems.

Abstract

We present VTMR, a two-stage framework for Video-To-Music Recommendation. In Stage~1, VTMR aligns comprehensive video and music signals in a joint audio-visual-text representation space and efficiently retrieves semantically compatible candidates using coarse global embeddings. In Stage~2, it reranks the retrieved candidates by attending to the temporal sequences of both video and music, thereby capturing fine-grained temporal correspondence. Evaluated on the video-to-music recommendation task, the multimodal retrieval stage improves R@10 from 14.2 to 15.9 and Median Rank from 75 to 58 over the strongest baseline; the temporal reranker further boosts R@10 to 18.3 and Median Rank to 46, demonstrating complementary gains from richer query encoding and temporal alignment. A human preference study confirms that VTMR is on par with a commercial baseline in overall preference, while outperforming a generative baseline in music quality.

Problem

• video-to-music recommendation
• multimodal retrieval
• temporal alignment
• semantic compatibility

Innovation

• multimodal retrieval
• temporal reranking
• video-to-music recommendation
• joint audio-visual-text representation
• semantic alignment