Managing bots on your CKAN sites?

CKAN Portal Operations
1

The topic of managing bots has come up in our work to maintain a CKAN instance, and was mentioned during several Monthly Live calls in the past year.

I recently came across resources from the Confederation of Open Access Repositories including results from a 2025 member survey and website with resources for managing the challenge of increased traffic from badly-behaved bots.

They list nine commonly-used strategies for managing the challenge. Hoping to start a discussion on this thread about your experiences with bots.

  • How has this issue impacted your work and your site?
  • If you’re managing a CKAN installation, which strategies have you found effective for dealing with the traffic?
  • Is there something you’ve tried that hasn’t worked as you had hoped?
  • Do you have any advice for others trying to manage this issue?
1 Like
2

Thanks for starting this thread and for the COAR link; that list of strategies is a great shared vocabulary. We’ve dealt with the same thing across the CKAN portals we run and just wrote up our approach in a two-part series, so I’ll keep this short and point there for details:

Part 1: Robots.txt, Crawl-Delay & CKAN Template Fixes (https://dathere.com/2026/07/managing-bots-on-our-ckan-sites-part-1/) - the biggest impact for us wasn’t rogue bots, it was CKAN’s own faceted search generating near-infinite tag/facet URL combinations. Disallow rules + crawl-delay in robots.txt, plus rel=“nofollow” on tag links and a conditional noindex, nofollow in the CKAN template itself, took Googlebot’s indexed pages on one of our sites from 152K down to under 1K in about 3 months.

Part 2: Rogue Scrapers & AI Bots (https://dathere.com/2026/07/managing-bots-on-our-ckan-sites-part-2/)- covers what robots.txt alone can’t stop (it’s a request, not a lock; a real share of bots ignore it entirely). We run Cloudflare Pro’s WAF + Super Bot Fight Mode for that layer, and for AI crawlers specifically, we’ve moved toward Cloudflare’s new Content-Signal directive (search=yes, ai-train=no, use=reference) instead of a blanket block - since for an open data catalog, the goal isn’t to disappear from AI systems, it’s to control how we show up in them.

Happy to go deeper on any part of this if useful to the thread.

3 Likes
3

Good stuff @sami-baig !

I remember distinctly Nadine’s CKAN Monthly Live presentation about the Humanitarian Data Exchange where they showed the spike in crawler traffic this year.

It’s the worst form of the “Tragedy of the Commons” - the whole mission of opendata is to publish data without encumbrance, but the scrapers/bots are abusing the resource - it’s affecting availability, and worse, costing portal operators bandwidth and storage overages.

Ironically, a lot of these bots are operated by the very same companies using the open data to train their AI - but the way they’re blindly crawling the portals is “dumb.”

That’s why I’m advocating for a “smarter” way to harvest data from portals - and that’s the key word - “harvest.”

Portals were all about distributing data from the beginning, but instead of naively scraping the site - they can just grab all the metadata from the catalog at the data.json endpoint - which was purpose-built for this very task.

For example, WPRDC’s catalog of 369 datasets has very rich metadata. Instead of crawling the site to scrape the data, you can get the very same data in a reliable manner from one endpoint - https://data.wprdc.org/data.json.

I already floated the skeleton of a proposal at the MLCommons Dataset working group. I’ll publish it here for comment, and perhaps, we can petition the big spider/bot operators to use the data.json endpoint instead.

2 Likes