Overview
We present a benchmark dataset specifically designed to evaluate matching systems using controlled vocabularies from the digital humanities (DH). This dataset includes manually compiled gold standard alignments for eight DH test cases, addressing DH-specific challenges such as multilingualism, specialized terminology, and the use of SKOS (Simple Knowledge Organization System) as a data model. The dataset, including the reference, is publicly and persistently available and incorporated into the OAEI 2024.
To obtain a high-quality dataset, we developed requirements including criteria for resource selection and present their practical implementation. By focusing on test cases that closely reflect real-world vocabularies, we facilitate advancements of matching systems, especially for subsequent mapping and integration tasks.
Evaluating the dataset using OAEI systems revealed significant weaknesses in their handling of SKOS and multilingual data, which shows the significance of our dataset. The evaluation also highlights the dataset’s quality, validity, limitations, and lessons learned, offering valuable insights for future benchmark development.
- Authors: Felix Kraus, Nicolas Blumenröhr, Germaine Götzelmann, Danah Tonne, Achim Streit
- Institution: Scientific Computing Center (SCC), Karlsruher Institut für Technologie (KIT)
- Year: 2025
- Conference: 19th International Workshop on Ontology Matching (OM/ISWC 2024)