T-CASP: Time-Aware Continual Aspect Semantic-Driven Incremental Pretraining

With the rapid advancement of medical foundation models, their deployment in clinical practice is increasingly required. However, privacy constraints of hospital-specific data make large-scale retraining infeasible, limiting model adaptability. To address this issue, we propose a Continual Aspect Semantic-driven Incremental Pretraining (CASP) framework, which enables efficient adaptation of foundation models to private data, and the pre-trained models can be effectively applied to downstream tasks. In this paper, we focus on fundus fluorescein angiography (FFA) in ophthalmology as a representative application scenario to validate the proposed approach. FFA is a critical imaging modality for retinal disease diagnosis, as it is able to capture dynamic vascular changes across multiple angiographic phases. However, most existing learning-based methods treat FFA images statically and independently, failing to exploit the rich temporal and phase-specific semantics that are essential for accurate diagnosis. In this paper, a Time-aware Continual Aspect Semantic-driven incremental Pretraining (T-CASP) framework is proposed for FFA sequences. To compensate for limited temporal descriptions in clinical reports, large language models are first used to construct a temporal disease knowledge dictionary with phase-specific semantic descriptions. Based on this dictionary, a disease correlation matrix is injected into contrastive learning to guide fine-grained image–text alignment. A multi-layer gated residual adapter is further designed to capture phase-level semantics and enable knowledge transfer across learning stages through phase-wise continual pretraining. Extensive experiments demonstrate that T-CASP effectively models dynamic temporal semantics in FFA sequences, yielding consistent performance improvements over time-unaware and static baselines in retinal disease recognition. By explicitly integrating phase-wise temporal knowledge and continual semantic refinement, T-CASP provides a clinically consistent and effective solution for temporal FFA analysis, enhancing robustness and diagnostic accuracy in ophthalmic multimodal learning.