TLDR: Researchers introduce Paper2Video, a benchmark of 101 scientific papers with author-created presentation videos and new evaluation metrics. They also propose PaperTalker, a multi-agent framework that automatically generates academic presentation videos from papers, integrating slide generation, subtitling, speech synthesis, talking-head rendering, and cursor grounding. PaperTalker produces high-quality, informative videos comparable to human-made ones, significantly reducing production time.
Creating engaging academic presentation videos is a time-consuming and labor-intensive process for researchers. It often involves many hours dedicated to designing slides, recording narration, and meticulous editing, all for a video that might only be a few minutes long. This challenge is compounded by the unique characteristics of academic content, such as lengthy research papers, a mix of text, figures, and tables, and the need to perfectly synchronize elements like slides, subtitles, speech, and a human presenter.
Addressing these significant hurdles, researchers have introduced a groundbreaking new framework called Paper2Video. This initiative tackles two core problems: how to effectively create presentation videos from scientific papers and how to accurately evaluate their quality. The project introduces the first comprehensive benchmark, also named Paper2Video, which consists of 101 research papers, each accompanied by author-created presentation videos, original slides, and speaker information. This rich dataset allows for robust testing and comparison of automated video generation methods.
To properly assess the effectiveness of generated presentation videos, the Paper2Video benchmark also includes four specially designed evaluation metrics. These are Meta Similarity, which measures how closely generated content aligns with human-made versions; PresentArena, where an AI acts as an audience to compare videos; PresentQuiz, which tests how well a video conveys the paper’s information through a multiple-choice quiz; and IP Memory, designed to gauge how well an audience remembers the authors and their work after watching a presentation.
Building on this foundation, the team developed PaperTalker, the first multi-agent framework specifically designed for generating academic presentation videos. PaperTalker streamlines the entire process by integrating several key modules. It begins with sophisticated slide generation, using LaTeX code for academic suitability and incorporating a novel “Tree Search Visual Choice” method for refining layouts and ensuring visual appeal. This method systematically explores layout variations to select the optimal design, addressing the common issue of text overflow or poorly sized figures.
Beyond slides, PaperTalker handles subtitling and cursor grounding. It generates subtitles for each sentence and then precisely aligns a virtual cursor’s movement with the narration and on-screen content, guiding the audience’s attention. For a personalized touch, the framework synthesizes speech using text-to-speech models, mimicking the author’s voice, and renders realistic talking-head videos of the presenter. A notable innovation is the parallelization of video generation across individual slides, which dramatically speeds up the production process by more than six times, making it highly efficient.
Experimental results on the Paper2Video benchmark demonstrate the superior performance of PaperTalker. The presentation videos it produces are more faithful to the original content and more informative compared to existing methods. In user studies, the quality of PaperTalker’s videos was rated as comparable to human-made presentations, and it even surpassed human-made presentations in PresentQuiz accuracy, indicating better information conveyance in shorter durations. The inclusion of a visible cursor and the refined slide layouts were found to be crucial for these improvements, enhancing audience engagement and knowledge retention.
Also Read:
- Decoding Event Transitions in AI Video Generation: The Critical Role of Timing and Model Layers
- The Evolving Landscape of AI Evaluation: From Simple Recognition to Complex Reasoning
This work represents a significant step towards automating the creation of high-quality, ready-to-use academic presentation videos, freeing researchers from a laborious task and potentially transforming how scientific findings are communicated. You can find more details about this research paper here: Paper2Video: Automatic Video Generation from Scientific Papers.
