From Sleep Staging to Spindle Detection: Evaluating End-to-End Automated Sleep Analysis

The paper presents a clear logical flow, starting from the problem of manual scoring limitations and variability, proposing an automated pipeline, evaluating its components and overall performance against human experts, applying it to a specific research question, and discussing the results and implications.

Strengths: Uses state-of-the-art deep learning models (RSN, SUMOv2)., Evaluates model performance against multiple diverse datasets, including those specifically designed for inter-rater agreement analysis (DODO/H, DREAMS, MODA)., Compares model-expert agreement not just to a single expert, but to distributions of human inter-rater agreement., Clearly defines evaluation metrics (Macro F1, IoU-F1) and their calculation methods., Statistical tests are used to compare group differences., Pre-processing steps are described in detail.
Weaknesses: The primary dataset for replication (BD) was annotated by a single expert (with verification), potentially limiting the generalizability of the replication success., Lack of explicit artifact handling is mentioned as a limitation by the authors., Quantitative discrepancy in absolute spindle densities between automated and original expert analysis is noted, although discussed.

The claims are well-supported by quantitative results presented in figures and tables, comparing model performance to human agreement levels and demonstrating replication of group differences in spindle density. The use of multiple datasets for evaluation strengthens the evidence.

The models themselves (RSN, SUMO) are based on prior work, though SUMOv2 is an enhancement described here. The novelty lies primarily in evaluating the feasibility and performance of an *end-to-end* automated pipeline on a clinical dataset and demonstrating its ability to replicate prior expert findings, and making the tools publicly available (including SomnoBot).

The demonstration that an automated pipeline can replicate complex clinical findings efficiently has significant potential to accelerate sleep research by enabling larger, more cost-effective studies. Providing open-source tools and a platform (SomnoBot) enhances this potential impact by making the methodology accessible.

Strengths: Formal and precise academic language is used., Concepts and methods are generally well-explained., Metrics (Macro F1, IoU-F1) and their calculation are clearly defined., The figures are well-captioned and integrated with the text.
Areas for Improvement: None