CeiTEA: Adaptive Hierarchy of Single Cells with Topological Entropy

Abstract Advances in single‐cell RNA sequencing (scRNA‐seq) enable detailed analysis of cellular heterogeneity, but existing clustering methods often fail to capture the complex hierarchical structures of cell types and subtypes. CeiTEA is introduced, a novel algorithm for adaptive hierarchical clustering based on topological entropy (TE), designed to address this challenge. CeiTEA constructs a multi‐nary partition tree that optimally represents relationships and diversity among cell types by minimizing TE. This method combines a bottom‐up strategy for hierarchy construction with a top‐down strategy for local diversification, facilitating the identification of smaller hierarchical structures within subtrees. CeiTEA is evaluated on both simulated and real‐world scRNA‐seq datasets, demonstrating superior clustering performance compared to state‐of‐the‐art tools like Louvain, Leiden, K‐means, and SEAT. In simulated multi‐layer datasets, CeiTEA demonstrated superior performance in retrieving hierarchies with a lower average clustering information distance of 0.15, compared to 0.39 from SEAT and 0.67 from traditional hierarchical clustering methods. On real datasets, the CeiTEA hierarchy reflects the developmental potency of various cell populations, validated by gene ontology enrichment, cell‐cell interaction, and pseudo‐time analysis. These findings highlight CeiTEA's potential as a powerful tool for understanding complex relationships in single‐cell data, with applications in tumor heterogeneity and tissue specification.