DeepSum: A Deep Learning Framework for Summarizing Animal Behavior

Abstract

The burgeoning field of ethology necessitates efficient tools for analyzing extensive video recordings of animal behavior, as manually sifting through hours of footage is both time-consuming and susceptible to observer bias. Here we present an innovative deep learning framework tailored for summarizing animal behavior videos, aiming to distill lengthy recordings into concise, informative segments. Leveraging the latest advancements in hierarchical video summarization, our approach employs a combination of Convolutional Neural Networks (CNNs) and Transformer models to extract and understand complex spatial-temporal patterns inherent in animal movements and interactions. The model is designed to recognize and prioritize key behavioral events, ensuring the retention of critical moments in the summarized output. Additionally, an attention mechanism is incorporated to adaptively focus on salient features, enhancing the model’s capability to discern subtle yet significant behavioral nuances. We assess our framework on a range of datasets containing different species and behavioral situations, and find that it outperforms current state-of-the-art techniques in terms of accuracy, coherence, and informativeness of the generated summaries. In addition to providing a consistent, objective method of analyzing animal behavior, DeepSum dramatically reduces the amount of manual labor needed for behavioral analysis, opening the door for advancements in ethological research and wildlife conservation.