A Multimodal Bone Stick Matching Approach Based on Large-Scale Pre-Trained Models and Dynamic Cross-Modal Feature Fusion

Among the approximately 60,000 bone stick fragments unearthed from the Weiyang Palace site of the Han Dynasty, about 57,000 bear inscriptions. Most of these fragments exhibit vertical fractures, leading to a separation between the upper and lower fragments, which poses significant challenges to digital preservation and artifact restoration. Manual matching is inefficient and may cause further damage to the bone sticks. This paper proposes a novel multimodal bone stick matching approach that integrates image, inscription, and archeological information to enhance the accuracy and efficiency of matching fragmented bone stick artifacts. Unlike traditional methods that rely solely on image data, our method leverages large-scale pre-trained models, namely Vision-RWKV for visual feature extraction, RWKV for inscription analysis, and BERT for archeological metadata encoding. A dynamic cross-modal feature fusion mechanism is introduced to effectively combine these features, enabling better interaction and weighting based on the contextual relevance of each modality. This approach significantly improves matching performance, particularly in challenging cases involving fractures, corrosion, and missing sections. The novelty of this method lies in its ability to simultaneously extract and fuse multiple sources of information, addressing the limitations of traditional image-based matching methods. This paper uses Rank-N and Cumulative Match Characteristic (CMC) curves as evaluation metrics. Experimental evaluation shows that the matching accuracy reaches 94.73% at Rank-15, and the method performs significantly better than the comparative methods on the CMC evaluation curve, demonstrating outstanding performance. Overall, this approach significantly enhances the efficiency and accuracy of bone stick artifact matching, providing robust technical support for the research and restoration of bone stick cultural heritage.