Hi Kannika,
Thank you for sharing these reflections. The questions you have raised resonate deeply with our own experience at Open Knowledge Nepal, where we recently launched AI integration on Open Data Nepal (https://opendatanepal.com), our civil society-led open data repository for Nepal.
We would like to share a few early learnings that speak directly to the questions you have raised.
On metadata and documentation for AI interfaces
In our experience, metadata is not only important for data discovery but also for data interpretation, including insight generation and aggregation. However, how useful it is depends heavily on how you implement the system. We used SQL queries so that all computation happens at the database layer, and the AI generates insights based on those query results rather than interpreting raw data directly. This gave us much more reliable and accurate outputs.
On what kind of metadata is most useful: we believe a dataset should have a clear, descriptive title and a plain-language description covering what it is about, what it contains, its source, and relevant context. Beyond that overview, a detailed schema is valuable. The resource should specify what tables it includes and, for each field, what the column represents, its data type (text, number, date, etc.), the units of measurement, and any codes or abbreviations used. This kind of structured information helps an AI understand and interpret the data more accurately.
Open Data Nepal’s CKAN API exposes exactly this through its resource_show endpoint, which returns a resource’s title along with its field-level schema, giving an AI the structured context it needs to read the data correctly.
On describing the limits of a dataset so that AI does not overstate what the data can support
We were also concerned about misuse of the AI feature, for instance, scripted or automated querying, or prompts entirely unrelated to the portal’s data, which can drive up costs unpredictably. To address this, we introduced two safeguards.
First, guardrails restrict the AI to questions about datasets hosted on the portal, so it does not respond to off-topic or out-of-scope prompts. This keeps the feature focused on its intended purpose and prevents it from functioning as a general-purpose chatbot.
Second, we introduced rate limits to prevent excessive use. Each user can ask up to 20 questions per day and make no more than 6 requests within a 5-minute window. These limits curb automated or abusive access patterns while still leaving genuine users ample room to explore the data. Together, these measures keep usage within intended bounds and make operating costs predictable.
On traceability and trust
Traceability was a design principle from the start for us. Every response the assistant generates is grounded in the actual datasets on the portal, not generic internet sources. But we have found that traceability alone is necessary but not sufficient for trust, especially among our primary users, who are not data specialists. Trust is built through familiarity and repeated accurate responses, not through understanding the technical architecture. This suggests that domain-expert review during the dataset preparation phase, not just at the point of AI output, is important.
On testing
We are still in early days on this, but our instinct is that real user questions are the most valuable testing input, precisely because they surface assumptions the system makes that no generic benchmark would catch.
Looking forward to following this thread and learning!