Back in 2023, there was an interesting exchange on the CKAN GitHub Discussions about how the CKAN platform should evolve to support DCAT 3. The conversation touched on everything from new DCAT entities and metadata models to implementation strategies, architectural implications for CKAN 3, and the evolving needs of different DCAT profiles.
Reading through that this morning, I was struck by a comment by @jqnatividad, who suggested that this might be precisely the kind of topic that would benefit from a dedicated Community of Practice as part of the POSE project. Since then, we’ve seen continued work on ckanext-dcat, including support for DCAT-AP v3 and DCAT-US v3, along with ongoing implementation work across the ecosystem.
Given that, with this forum, we now have a place to support the sorts of community of practice suggested by Joel some years ago, it seemed like a good opportunity to revisit that conversation here today.
For those of you working with DCAT, I’d love to know:
What aspects of DCAT have proven most challenging to implement in your organization?
Are there areas where you’d still like to see stronger support?
What experiences, examples, or resources would be most helpful to share with others working on DCAT?
@jze having started that original GitHub discussion a couple of years ago - and now with the benefit of a few more years of implementation experience in Schleswig-Holstein - I’d be especially interested to hear whether your thinking has evolved since then, or if the same questions still feel most pressing.
Thank you very much for bringing this up. I would have started a thread on this in the next few days as well. The DCAT metadata model ist very important, particularly as the EU uses it in more and more data related directives and acts.
In Germany, the transition from the OGD metadata model to DCAT was completed around 2018. So I was in the lucky position of being able to get started with DCAT straight away.
In my opinion CKAN is currently not very DCAT compatible. Therefore the EU data portal switched from CKAN to another software some years ago. The national German data portal has also already decided to do this.
I think the main issue is that DCAT is based on RDF. This brings the extensibility XML promised 25 years ago. You can have your common metadata and simply add your own domain specific metadata. These additions are not simple key-value pairs but can be arbitrary complex, even linking to resources somewhere else (e.g. topics in a thesaurus or a Wikidata item). And if all works well that metadata can be transported from the local data portal through the regional data portal, the national data portal, and finally to the EU data portal without any loss. At the moment, we are experiencing data loss whenever a CKAN-based portal is involved.
The other major issue is the lacking support for the three top-level metadata classes: Dataset, DatasetSeries and DataService. (The geospatial domain uses a similar three-part metadata model - I think DCAT took this over from them.) At present, only Dataset is supported. The information about a DataService is somehow stored in an extra field of a resources. But that’s not actually correct, as services are top-level elements independent of a Dataset.
I think both issues can be solved:
The original RDF could be stored in the database. It would be even possible to use a JSON typed database column for it if the RDF is serialized as JSON-LD.
A package already has a property type. That could be used to specify whether it is a Dataset, a DatasetSeries or DataService. You would then need three different templates for these three different types.
For the portals that are not “leafes” in the metadata system it is also important that the harvesting of DCAT catalogues works without data loss.
We have programmed a few workarounds so that we can handle DCAT metadata to some extent. For example we need a proxy that repairs the DCAT output before sending it to the national data portal. But that doesn’t feel right and is difficult to maintain.
Sorry that this has turned into such a long post, and one that sounds so negative. But I think we need to address the issues in order to solve them.
@jze no apologies needed! I really appreciate the time, care, and candor it took to write this.
One of the questions I posed in the original had to do with what kinds of experiences, examples, or resources would be most helpful for others working with DCAT - you’ve gotten right to the heart of that question. Given what you’ve shared, understanding where implementations converge and diverge feels like an important part of moving the conversation forward.
@jqnatividad one of the things that caught my attention in the original GitHub discussion was your suggestion that this topic might benefit from an ongoing Community of Practice. Reading Jesper’s response made me wonder whether creating more opportunities to exchange implementation experiences like this iis the sort of conversation you had in mind?
You are right in that CKAN’s DCAT support can have limitiations, the main ones you mentioned being:
Lossy metadata transfer between systems. There’s no way around that, CKAN is not powered by graph database and to leverage its features like search, validation etc the RDF needs to be translated to CKAN’s data model. That’s the default ckanext-dcat behaviour for consuming and generating DCAT RDF metadata. But that doesn’t mean it needs to be the pipeline for federating DCAT RDF across sites. Take for instance the example of the Spanish National Open Data Portal, which imports the RDF in parallel to a SPARQL endpoint, and reuses all original RDF triples when generating the serializations for the next harvesting level, thus preventing metadata loss. See this comment for more details, I’d happy to put you in contact with David if you want to learn more about their setup.
Top level metadata classes: `Dataset` is fully supported , and DatasetSeries as well, see documentation at Dataset Series - ckanext-dcat . `DataService` is indeed very rudimentary. I know that folks at Derilinx have implemented support for it in a couple of ways (e.g. custom group or dataset schemas) and I’m definitely keen on adding support for it in ckanext-dcat. If you have further ideas I’d be happy to discuss potential implementations.
DCAT compatibility can mean very different things, and I’d argue that the level of DCAT support that CKAN offers via ckanext-dcat is very high, certainly among open source data portals software and covers a lot of requirements for most portals in terms of compliance. Again, all this with the limitations of CKAN’s design, but given its flexibility I’m sure there are ways to work around them.