Multimedia Papers

@MultimediaPaper

New Multimedia papers from arxiv.org: multimedia. Thank you to arXiv for use of its open access interoperability.

Worldwide arxiv.org/list/cs.MM/new Joined March 2010

2 Following 100 Followers 5K Posts

Latest posts 5

DASWeave: Weaving Evidence across Speaking Conditions for Multimodal Depression, Anxiety, and Stress Screening

Binyan Chen, Tong Pan, Yadong Liu, Xin Yun, Shangfei Wang
arxiv.org/abs/2611.13544 [𝚌𝚜.𝙼𝙼]

Multimodal screening for depression, anxiety, and stress in children and adolescents uses verbal and nonverbal behaviour. Most existing multimodal methods learn from a single recording or interview. Yet a participant's behaviour changes with the speaking condition, and these changes hold evidence that is invisible in a single recording. We propose DASWeave to bring this evidence into a neural network that encodes each response and then combines all responses of a participant. We train supervised trees to find which behavioural statistics matter for screening, both within each response and in how they change across conditions. We feed the tree outputs into the network both before and after this combination. We fine-tune a large language model to relate what a participant says to how they say it across conditions, and feed its representation into the network after the combination. The language model reads behaviour as text descriptions. We therefore derive additional training targets for

Multimedia Papers @MultimediaPaper 20h

YTShorts-92K: An Interpretable Multimodal Dataset of YouTube Shorts

Celian Charrin, Sidonie Abuzakuk, Artúr Vlaszov, Nándor Birher, Roland Molontay
arxiv.org/abs/2611.13002 [𝚌𝚜.𝙼𝙼]

Short-form video has become one of the dominant formats of online content, yet large-scale datasets enabling the analysis of what makes these videos successful remain limited. Existing resources often focus on post-publication popularity signals or rely on non-interpretable representations such as high-dimensional embeddings. Here, we introduce YTShorts-92K, a dataset of 92,654 YouTube Shorts collected from 345 creators and spanning 12 manually defined content categories. Each video is enriched with an interpretable multimodal feature set covering metadata, textual information, visual characteristics from frames and thumbnails, and cross-modal consistency measures. We describe the collection methodology, data cleaning process, and feature extraction pipeline, followed by descriptive analyses of the diversity of creators, topics, and extracted features. This dataset offers a reusable resource for studying short-form video content, online attention, and multimodal content analysis.

Multimedia Papers @MultimediaPaper 3d

UniMore: Fine-Grained Supervision for Unified Understanding and Generation

Haoyue Bai, Yinyi Luo, Yicheng He, Qingsong Wen, Sharon Li, Jindong Wang
arxiv.org/abs/2611.08996 [𝚌𝚜.𝙼𝙼 𝚌𝚜.𝙲𝚅]

Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image–text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image–text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter

Multimedia Papers @MultimediaPaper 6d

DiaHeads: Jointly Generating Coordinated 3D Dialogue Heads with Non-verbal Cues from Dual Audio

Jiahe Wang, Yuxin Wang, Xin Yun, Binyan Chen, Tong Pan, Zhenjie Liu, Shangfei Wang
arxiv.org/abs/2611.06970 [𝚌𝚜.𝙼𝙼]

Audio-driven 3D head animation has made considerable progress in generation realism, yet generating coordinated behavior for two-person dialogue remains difficult, because current dialogue head generation methods overlook the modeling of non-verbal behavior semantics between the two participants. To address this limitation, we present DiaHeads, a framework that generates the behavior of both participants jointly. The two audio streams are concatenated and decoded together, so each participant's motion is conditioned on the other, which makes the interaction part of the generation itself. We further introduce four discrete non-verbal cues, gaze-on, head shaking, nodding, and smiling, and fuse them with the continuous parameters in a shared latent space through a product-of-experts posterior. These cues carry the interaction semantics and constrain how the continuous motion is generated. A flow-based prior over the shared latent variable replaces the standard Gaussian prior and makes the

Multimedia Papers @MultimediaPaper Nov 3

Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Xusen Hei, Jiali Chen, Jinyu Yang, Yi Cai
arxiv.org/abs/2611.02928 [𝚌𝚜.𝙼𝙼]

Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.