Deciphering the protein landscape with ProtFlash, a lightweight language model

Protein language models have been gaining attention and have achieved some exciting breakthroughs in protein modeling tasks compared with the utilization of co-evolutionary and biological priors. To overcome the shortcomings of existing large-scale protein language models, such as high computational complexity and large memory consumption, we propose a lightweight protein language model, ProtFlash. It is the first protein language model with linear complexity based on an attention strategy, which differs significantly from existing methods in that it primarily utilizes multiple positional encodings and a mixed chunk attention mechanism that combines local and global attention. Furthermore, the results of the Tasks Assessing Protein Embeddings show that ProtFlash achieves a better or equivalent performance than the state-of-the-art protein language models. As shown through a rigorous comparison of computational complexity and memory consumption, it also performs better than existing protein language models.