My name is Adam and @mushon and myself are leading obudget.org - the Israeli Open Budget project. We are really thrilled to join this forum!
We’re currently looking into using OpenSpending as a backend for our obudget site.
One of our biggest unknowns is whether OS API supports the different data kinds (and irregularities) we have here.
Currently we have (and we’re obviously planning to add more):
Yearly budget articles (planned, revised and executed) - each one defines multiple monetary values as well as personnel limits etc.
Changes being made to the budget throughout the year, and related metadata (e.g. each set of such changes has its own explanation letter that our Ministry of Finance produces)
Government Grants - yearly grants, given under a specific budget article
Procurement Tenders - here we also have extra meta data, such as tender documents etc.
(you can take a look at our current api here: מפתח התקציב )
We would very much like to hear your input as to whether and how OS can be integrated wit obudget.org.
Hi adam! I was talking this through with @mushon a few weeks ago in Berlin, as I was then trying to work out how to make OpenSpending’s data import more flexible and to support more messy and diverse input data.
Unfortunately, that effort has been shafted by OKF and the branch I was working on is now deceased, so some of the things that I mentioned to @mushon won’t make it into the platform.
@pudo the fact that you didn’t get what you wanted in terms of tech setup for OpenSpending doesn’t mean you need to start trolling (I’d also note that OSEP 1 in no way precludes use of Flask of other contributions).
Let’s keep the discussion constructive and focused on what the OpenSpending project can do for people.
It is worth mentioning, though, that “what I wanted” was exactly that: a constructive discussion about the technical changes which I proposed and prototyped. So I’m not the only one
ok guys, glad to see some passion infused to get this thing going…
We really are interested in having oBudget.org function as a part of a larger community and to have our development resources be something that parallel initiatives could take advantage of. I am sad to hear that the initiatives proposed by @pudo have been terminated, @rufuspollock can you or anyone else here add some details on what can OpenSpending do to address requirements such as those surfaced by adam for oBudget? It would be useful for us to know as we have stalled the development of some oBudget backend features hoping to integrate and then develop them on top of the OpenSpending platform.
UnTrollishly yours,
Mushon
These are great news. I do hope we can make this work and I think we can. There is a big opportunity for you to, as you said, have your development be something that similar initiatives can take advantage of and for you to lead the way.
OpenSpending is transitioning from a single system where you import your stuff in the hopes of getting something that works for you out. The grand vision is to have OpenSpending as a group of integrated services that devs like you can tap into or extend. So if you’re interested in using the validator @pwalsh has been building, clean and transform the data into a standardised format and uploaded into a central repository you can then provide your API on top of it.
Now is the perfect time for you to join because you can then help us figure out what standardised format would be best for the list:
Yearly budget articles (planned, revised executed) could be tackled by Budget Data Packages but it seems like you have more monetary values (and net+gross for each one).
Changes being made to the budget throughout the year, can be tracked with Budget Data Packages as well (they’re just Data Packages with special budget metadata (and the metadata is on a resource basis so you can have multiple resources). The related metadata is interesting (including the letter etc. which we would need to look into).
Government grants could be covered by a budget data package, based on whether it is a the plan or the paid ones, however it might be wise to introduce a special grant type to make it clear).
Procurement tenders is something we haven’t looked at in detail but it’s been on the horizon. So this is an opportunity to help each other out as well
I’m really excited about this. There’s a dev meeting tomorrow on our freenode IRC channel: #openspending (one at noon GMT+0 and another one at 11pm GMT+0). Could you join and we can then have a discussion? The main topic will be the importing and storage of data so it fits this discussion perfectly.
@adam and @mushon: it is not simply a matter of what Open Spending can do for you, but what you can do for Open Spending :).
As should be clear, Open Spending is transitioning to a micro services model.
A goal of this vision is that projects such as yours are able to tap into the Open Spending ecosystem without a complete buy-in to the whole thing as a monolith.
For example, use the data importing flow to create and validate budget data packages; use the low-level file storage API as a common base to build a more specific API on top of, etc.
For me, dealing with municipal data in Israel (and some initial directions in Australia), comparative analysis (across time, across entities) is critically important. I know that it is in your project too. AFAIK, the current Open Spending API does not allow for that. But we can get it there if we all focus our energies on it by hacking around the same ecosystem.
We are pretty unique in the Public Knowledge Workshop (for those that do not know, that is a place in Israel where I and @adam and @mushon volunteer to hack and design on open data projects) in that the obudgets codebase, and the one I built for muni budgets, have been built from ground up to support use cases that OS has not been able to support, and therefore there is a chance to bring those use cases into to the discussion of OS right now.
The benefits should be clear: a codebase with a global community; data standards for storing and accessing budgetary data, etc.
No, on the contrary. This opens up even more possibilities to visualise the data. One of the things we will have to work towards is conformed data (similarly structured), i.e. data where we know what we can expect from each dataset. That means we can make visualisations even easier and do more awesome things like compare datasets in a better way and more.
We (@shayshei and me) are the two who tried to make a POC of openspending and open-budget.
We encounter the following issues:
We created a basic project- https://openspending.org/obpoc/ and loaded the following CSVs-
– the.open-budget.org.il/api/budget/00/2010/kids?o=csv&limit=22
– the.open-budget.org.il/api/budget/00/2013/kids?o=csv&limit=22
This resulted the following error msg:
(IntegrityError) insert or update on table “obpoc__entry” violates foreign key constraint “fk_obpoc_time” DETAIL: Key (time_id)=(1) is not present in table “obpoc__time”. ‘INSERT INTO obpoc__entry (id, code, title, time_id, amount, net_revised, net_used) VALUES (%(id)s, %(code)s, %(title)s, %(time_id)s, %(amount)s, %(net_revised)s, %(net_used)s)’ {‘code’: u’0000’, ‘title’: u’\u05d4\u05db\u05e0\u05e1\u05d5\u05ea \u05d4\u05de\u05d3\u05d9\u05e0\u05d4’, ‘amount’: 325287959.0, ‘time_id’: 1, ‘net_revised’: 325287959.0, ‘net_used’: 316528481.0, ‘id’: ‘f5dd70cc00d9db6d731a8639d40c6de91f2d822f’}
We have empty cells in our tables, and therefore we can’t load all our data.
We wanted to map all the rest of our fields, but the constraints (i.e must have an “amount” field) makes it difficult. The table we wanted to map is: Login to Redash
We want to understand if openspending can support our scheme, and if it’s not the case right now - where the changes should be made (our side or yours). Can you make a example of an openspending project which supports all the fields of that table?
It is great to see you tried to do an upload. It looks like you haven’t made any attempt to conform the raw data output from OBudget to the schema that Open Spending requires.
(I’d also note that 