OpenSpending Next: Status Update


#1

OpenSpending Next is coming along well, and we want to update the community with where things are, and what the coming weeks and months have in store.

Technical leadership

Firstly, I’d like to announce publicly that Adam Kariv ( @akariv ) is the Technical Lead for OpenSpending.

Adam is a highly-skilled developer with loads of commercial experience, and additionally, many years of experience working on open fiscal data projects in Israel. His Budget Key is an innovative and widely used open data platform providing insight into the state budget in Israel.

Adam has been with us at Open Knowledge International for several months, and officially took over the Tech Lead role on OpenSpending in February. We are very excited to have Adam on board.

My role in OpenSpending is now that of technical “Product Owner”, so I am still very involved, especially around feature design and working on new documentation.

The team

There are several of us involved in the day-to-day work around OpenSpending, so let me also take this opportunity to introduce everyone:

  • Cecile ( @CecileLG ) is the Project Manager.
  • Adam ( @akariv ) is the Technical Lead, and currently focussed on the API, as well as data interfaces to push OpenSpending forward as a tool for deep analysis of fiscal data.
  • Dan ( @danfowler ) is the curator of Fiscal Data Package, the data specification on which OpenSpending Next is built.
  • Levko ( https://github.com/kravets-levko ) is a working on the frontend UI and logic of the Packager and Viewer apps.
  • Igor ( https://github.com/borysyuk ) is a working on the frontend UI and logic of the Packager and Viewer apps.
  • Sam ( @sam ) is the user interface and interaction designer.
  • Evgeny ( https://github.com/roll ) has written code around Datastore access and the CLI in Python for working with Fiscal Data Package and the OS Datastore.

Communication channels

You can contact the team at anytime via these channels:

  1. Right here, in the OpenSpending discussion forum
  2. On Gitter.im in the OpenSpending chat room
  3. By opening issues on the OpenSpending issue tracker

What we’ve achieved

Since we started on OpenSpending Next, we’ve been hard at work in the following general areas:

  • Working with our partners at GIFT and BOOST to develop Fiscal Data Package
  • Implementing the core components of the OpenSpending Next MVP:
    • OS API: A implementation of A V3 API based almost entirely on the excellent Babbage by @pudo. This API also features a fully backwards compatible implementation of the OpenSpending V2 API for legacy apps that depend on it.
    • OS Conductor: A set of handlers for authentication, authorisation, and web hooks across the components.
    • OS Viewer: A workspace to visualise data packages loaded into OpenSpending. The whole viewer can be embedded in 3rd party apps, as well as specific view components. The view components use a refactored codebase of Babbage.UI by @pudo, and include some new views like a GeoView.
    • OS Packager: A guided wizard to create Fiscal Data Packages from raw source data, and upload them straight to the OS Datastore.
    • OS CLI: A command line interface to validate Fiscal Data Packages, and to push them to the OS Datastore.
    • OpenSpending: A Docker Compose setup which pulls together the components into a singular installable platform.

There are a lot of components and pockets of activity involved in building OpenSpending Next, as demonstrated by the above list! We are now at a critical time of bringing all the components together, and implementing user interfaces for public alpha testing.

Next steps

As announced here, the current OpenSpending.org has been in read-only mode since mid-January, and we estimated that it would stay this way for 2 months.

At this point, we are not ready for public use of the OpenSpending Next alpha, so the read-only period will last for several more weeks.

Our current target is the first week of April. We are working hard right now to make this date. In broad strokes, what we are working on in the weeks til then are:

  1. The UI for the Packager
  2. The UI for the Viewer
  3. Edge case bugs in the data modeling and loading flow
  4. Documentation for developers, data contributors, and end users

Once we are satisfied with these areas, and not before, we will make a public announcement that the alpha release is ready for testing. The expected steps for there are:

  1. Work with some data contributors to iron out bugs in the flow from raw data -> API -> Views on data.
  2. Fix those bugs, and any additional user experience issues.
  3. Announce the beta release of OpenSpending Next.

Towards closing the current OpenSpending API

From the beta release onwards, all data processing and APIs are expected to be stable for use, and we’ll start approaching the various consumers of the current OpenSpending API about migrating to the new API.

While we are still in beta, we will still keep the current OpenSpending API live, but we will announce a deprecation period in which all consumers should move to the new API, and finally we’ll transition out the old codebase completely.

Looking forward

The MVP of OpenSpending Next implements the core of our vision for OpenSpending:

A modular architecture, a flat file datastore based on a clear and explicit data description specification, and the start of an new ecosystem of tools and workflows around fiscal data. We have many more things planned, which we’ll discuss in detail after we get the alpha released.

We can’t wait to ship OpenSpending Next, and get feedback from the community, and we thank you for your patience while we’ve been deep in the trenches working on the low level aspects of the technical product.


Open Spending Next : Update and Teaser
When is next OpenSpending Community Hangout and Tech Update?
OpenSpending.org moving to read-only mode
#2

Fantastic and significant enough that I would recommend posting on community.openspending.org :smile:

Well done to the team - this is a major, major upgrade and progress has been really solid!

Thoughts:

  • Re next steps I think it is valuable to have a specific commitment on a “go-live” time. Even if it turns out this has to be pushed making a promise “out loud” has power whereas an undefined commitment to some time in April does not.
  • I also wonder whether different parts could be released incrementally. E.g. could “power users” start using and pushing to the backend right now even if the entire workflow is not yet fully wired together? (Also where can I find any info on the location of the Data Store and its structure?)
  • (Super small) Look forward to team having profiles on the OK directory so we can just @ them here … :wink:

#3

Excited to start showcasing the new functionality, how goes the progress?