Parallel web crawling for customer analytics

This paper presents a parallel web crawling system designed to collect publicly available customer shopping data from a retailer's website, aiming to understand customer purchase behaviors and to formulate customized engagement strategies. The system architecture includes a parallel crawling framework with scalable leaf units that facilitate the distributed crawling process, expediting the complete downloading of the retailer’s product webpages. The collected data encompasses customers’ purchase data, including product name, price, and date, alongside demographic information such as gender, age group, and location. Our dataset comprises 836,369 shopping records, representing 27,160 items purchased online by 455,088 customers over a decade-long period. This dataset can be transformed into recency–frequency–monetary ( RFM ) metrics, suitable for k -means clustering analysis to segment customers into distinct clusters: Lost , Potential , Hibernating & Valuable , Extant & Active , and Loyal & Lucrative . These clusters provide valuable customer profiles to assist the retailer to meet the unique needs and preferences of each cluster. Demographic analysis of the clusters reveals that female customers constitute nearly 90% of the Potential , Extant & Active , and Loyal & Lucrative groups, underscoring their significant role as active and valuable consumers on the website. Furthermore, customers aged 50 and above account for over 62% of the demographic across all clusters, highlighting the retailer's appeal among older shoppers. A geographical breakdown shows that California, Texas, New York, and Florida are the states with the highest customer concentration in every cluster.