Dataset Similarity Detection for Reuse Protection in Federated Data Spaces with Privacy Considerations

Federated data spaces, established through initiatives such as IDSA and GAIA-X, enable organizations to share and monetize datasets under contractual terms. However, enforcing these contracts—particularly detecting unauthorized reuse or modification of datasets—remains an open challenge. We present the Off-Platform Contract Inspector, a component of the PISTIS framework, that implements a modular similarity-detection pipeline combining path-value Jaccard similarity, field-aware type-specific comparisons, and sentence-embedding-based semantic analysis across structured, semi-structured, and unstructured datasets. This contributes as follows: (i) an Inverse Document Frequency (IDF)-weighted structural similarity mechanism that discounts common domain vocabulary via Inverse Document Frequency weighting over the data space catalog, combined with a schema-evidence-gated fusion that reduces false positives from domain vocabulary overlap; (ii) an adaptive threshold optimization mechanism that learns modality-specific fusion weights and decision thresholds via cross-validated grid search; and (iii) a privacy-preserving similarity layer based on MinHash Locality-Sensitive Hashing signatures, Bloom filters with OR folding alignment, and Laplace noise for differential privacy, enabling cross-organizational dataset comparison without exposing raw data. Further, we contribute a threat taxonomy of seven dataset modification types ordered by detection difficulty, and evaluate the system on dataset pairs derived from real-world datasets across three smart-city application domains (Mobility, Energy, Automotive), with controlled augmentations applied to model adversarial behaviors. The IDF-weighted pipeline achieves high precision on intra-domain hard negatives—pairs of different tables from the same data space that share domain vocabulary—where text-similarity baselines produce false positives. The adaptive scheme learns per-modality fusion weights via cross-validated grid search. The privacy-preserving mode operates without accessing raw data and runs noticeably faster than the full pipeline, enabling screening while preserving data confidentiality.