Domain generalization and punch mark classification

• Convolutional neural networks show capacity to classify punch marks. • Classification often fails for paintings not in the training data. • Several domain generalization methods appear to offer little improvement. • Punch mark classification gives insight on domain generalization strategies. Punch tools were used to mechanically make decorative impressions—called punch marks—on gold ground paintings, becoming particularly widespread in Italy during the 14th and 15th centuries. Punch tools were frequently reused for multiple paintings, sometimes from different workshops, and the impressions they leave behind support a variety of art-historical investigations by evidencing workshop practices, attribution, contemporary connections, and more. In particular, classifying punch marks in paintings enables comparisons with extensive indices that were created by art historians/conservators Erling S. Skaug and Mojmír Frinta over the course of several decades. In this paper, we explore the potential of automated methods for punch mark classification. As in most image analysis tasks, deep neural networks are state of the art. Indeed, convolutional neural networks can produce highly accurate classification results, but often falter when confronted with images from paintings not represented in the training data. This is a particularly relevant problem in cultural heritage applications such as punch mark classification, where the size of training sets is typically small. For this reason, we have explored domain generalization methods, which aim to maximize accuracy on some target domain (images from a painting unseen in training data) using various source domains (images from paintings used for training). We find that, despite their promise, domain generalization methods (with explicit domain labels) unfortunately often offer little advantage over baseline convolutional neural networks (without explicit domain labels). Our results provide insight on the capacity and limitations of several off-the-shelf deep learning methods to automatically classify punch marks. Conversely, the structure and the challenges of punch mark images provide an interesting test case for insight on domain generalization methods.