When we asked folks about what they wanted to discuss on this forum, metadata quality was one of the more commonly raised topics. I’ve been thinking on this, and wanted to ask you all about what “quality” means to you.
I come from a background in research data management and have worked with colleagues at Metadata Game Changers to explore metadata quality and consistency across the global research infrastructure. The two principles there, Ted Habermann and Erin Robinson, have since done some interesting work examining metadata quality from the perspective of the FAIR Principles. One of the ideas I appreciate from their work is the observation that metadata quality isn’t necessarily about having more metadata, but more specifically it’s about having the metadata needed to support particular use cases.
For some organizations, quality may mean richer descriptions that improve discovery. For others, it may mean consistent controlled vocabularies, persistent identifiers, provenance information, accessibility, interoperability, or something else entirely.
So, I am curious, what does “metadata quality” mean to you and how do you know when you see it?
Thanks, @jze! I was loosely aware of the EU’s Metadata Quality Assessment methodology, but hadn’t had the opportunity to explore it in much detail before (so I doubly appreciate you sharing it).
One thing I particularly appreciate about the EU methodology is that it takes such care to define the terms that comprise the methodology before moving on to identify dimensions and indicators.
How different organizations define the terms by which they assess metadata quality was actually part of what I was hoping to explore with my earlier prompt. Different organizations, communities, and jurisdictions often have different priorities, use cases, and expectations, which means they may also define metadata quality somewhat differently. I’m curious to hear whether others are working from similarly explicit methodologies, or whether metadata quality is understood more informally within their own organizations.
We’ve been doing a lot of work on automating FAIRification in several of our projects.
I’m afraid to report that in all the CKAN implementations we’ve been involved in the past ~15 years, metadata upkeep leaves something to be desired.
And it’s not unique to CKAN - in a 2021 Usability Study the Texas Water Development Board commissioned BEFORE they started building their Water Data Hub - where they interviewed 10 portal operators using various platforms (CKAN, Socrata, ESRI, etc.) - the main complaint of portal operators was metadata upkeep.
This led to the development of DP+ where we created the Data Resource Upload First (DRUF) workflow - where we turned the CKAN data registration/ingestion process upside down.
Typically, when you register a new dataset in CKAN, you fill out the metadata and then upload the data - the assumption being, you already prepared all the metadata beforehand.
With the DRUF - you upload the data first, and then thanks to the blazing speed of qsv - we process all the data in a few seconds - extracting dozens of statistical measures and computing FAIR metadata suggestions asynchronously that by the time the CKAN user goes to the metadata entry pages - metadata suggestions are already in place.
We implemented this by extending ckanext-scheming extension so the metadata schema can have user-definable Jinja formulas. Among these suggestion formulae is the ability to even ask an LLM to draft Descriptions, Tags and an extended Data Dictionary.
Already, among the formulas we bundle are DCAT3 optimized ones that can compute DCAT properties that are non-trivial to compute manually - e.g. spatial extent, periodicity, spatial range, etc.
Some of the most interesting work about quality in general was the ISO 19157 Data Quality standard. Instead of defining a standard way to measure quality, say some 1-10 scale, the standard focused on describing what quality measures and procedures were used to measure data quality. It also included some evaluation of usability that I think is important. IMHO, if users can understand and trust the data using existing metadata, it is quality metadata, i.e. users must be included in the assessment…
My work focuses on three measurable elements of metadata quality: completeness (number of records with an element / total number of records), connectivity (number of identifiers / number of possible identifiers), and consistency (consistent representations of elements, i.e. names, flags, etc.). If you have metadata in DataCite, these tools (https:/bit.ly/RepositoryTools) might help you measure and understand your metadata in terms of these measures. There are also guides for repositories ( Repository Guide — DataCite metadata tools ) and consortia ( Consortium Leader's Guide — DataCite metadata tools ) for using the tools.
I have not seen this before - thanks for pointing it out…
I like that it provides specific metadata elements and scoring. That is very similar to my approach (Metadata Game Changers — Metadata Tools) although my weighting is uniform. It is also interesting that if you go to this page and download the report as a spreadsheet, it provides current scores for over 200 repositories (none in US). Six of these have an excellent rating. I am very interested in identifying good examples, bright spots, across groups of repositories ( DataCite Bright Spots – Repositories, Consortia, and Improvements — Metadata Game Changers ), shedding some light on their successes, and using them to motivate improvements across the network.
It is unfortunate that these checks do not include any identifiers. I think those are increasingly important and also an area with plenty of potential for improvement.