https://github.com/mcnabber091/Econ-Factbook is this what you mean? I wasn’t able to make it for Open Economics yet.
Yes. Just transfer ownership to OKFN-OE and then we will change the repo’s origin link in the terminal directly to the organization repository
I just got the message, ‘You don’t have admin rights to okfn-oe’
Do you mind trying again? Forgot to tick one important checkbox
Ok cool it’s on there.
Okay, now things are in place, I believe.
Feel free to add whatever you want/need to the website folder (I named it as econfactbook.org) and please change the readme for something you feel that suits best.
Thanks
Thanks, this looks good so far. I like most of the readme page, I will make edits as I see fit.
Hi Gustavo,
thanks for the e-mail update on this. I may soon have some time to help out with this a little. Let me know what you need.
Rodney
The theoretical information is on the forums. The rest is on GitHub, which
you can find the links in the forum as well. If you face any issues, let me
know
Ok have had a brief look at the Python code on github in the discussion above it mentions using R for scraping have done a bit of that. Let me get up to speed and will get back to you all here.
Hi again,
Hum, yes. Initially I was going to propose us to use R, but Python is kind
of more universal and it is a language that I am putting some effort to
learn. I’m currently more comfortable using R for this kind of tasks, but
Python is fine. I’m not sure though, if Python is a good language for
scrapping. Judging from what I’ve read online, it seems it is possible,
although it is not the focus of the project. As far as I remember, you’re
good with it, so perhaps you can help in that end. If that’s the case, then
I will prepare a script in R just to get the datapackages and, hopefully,
this will allow the project to move on. If not, I’m really looking forward
to work hard in Python.
Also, @alexpeek1, I’ve had no luck finding any designers for you/us. In
the first Saturday of each Month, I know a meeting that takes place here,
in Porto, Portugal, and I will attend it next time to talk about Open
Economics and your project. Maybe I’ll have better luck in there.
Best Regards / Obrigado e com os melhores cumprimentos,
Gustavo Silva
I added a list of statistics we should target on the wiki: https://github.com/okfn-oe/Econ-Factbook/wiki. I will soon add links to sources showing where the data is located. Feel free to edit this list if you think there are other economic statistics you think we should get.
Gustavo, good luck with your meeting in Porto. I would be interested to hear how that goes.
Rodney, glad to see you might be interested to help out. If you have any questions let me know.
Best regards,
Alex
Hi all, just quickly read back through the thread, seems like an interesting project. Glanced at your github project and, if I may, can I suggest the world bank data API as a good way to get up to date data. Instead of scraping a website set up for human readers, you can get structured data directly with a link like this http://api.worldbank.org/countries/all/indicators/SP.POP.TOTL?date=2014:2014&per_page=20000&format=json. This at least works with world bank data, but you may need to follow the other approach for other pages.
You could either follow the python approach and scrape data then save it on your server, and set up the scraper to run periodically so that data remains up to date, or you could get the data as you load the page each time (using Javascript or PHP perhaps). I tried out a project doing the latter which is a bit clunky but you might find interesting.
I’ve also done some python scraping of websites with less structured data, and am happy to contribute to your scripts if you would like. Let me know if I can help with anything.
Cheers!
Thanks @alexpeek1!
I believe that will be very useful!
@mcallaghan, the work group is open. Everyone is free to contribute to it! Feel free to contribute whenever you’d like. That’s why I created a GitHub organization for the group: People can see what is the status of things, can read a bit more about on the forums, and then just fork, do their thing and post it again to GitHub. If needed, we can discuss changes online
Your project is pretty much what I’d like to do for Econ Factbook. Maybe we can merge ideas and put it all together in one place? The Python script I’ve been trying to work on is just to scrap data from the sources Alex used. From there, I believe creating a page in JavaScript is more design-friendly, more beautiful and also more appropriate to build something young, easy to read and scalable, like your project.
I will create a page in the Wiki as a reference to what is this project - maybe even a summary of what has been discussed here.
Hi @gsilvapt.
Cool, that link is just sitting around doing nothing. If you can think of useful ways to use it, and a place to put it, I am happy to contribute.
We’ve been collaborating on GitHub, but I believe It would be interesting to add your map to the original website, to later be published via the group’s website. Do you think you can get in touch with @alexpeek1 here to see if you can do something about it? I believe you two can add something to a new group of folders under /site (to be created)?
Thanks for collaborating guys!
We’re getting there! At least all data that comes from World Bank is solved - or will be in the future, as I am not bashing my head with Python and APIs…
@gsilvapt Good news to hear that the World Bank data is going well. Thanks for your work so far. I have added source-links here. I hope this can be helpful for retrieving the data.
@mcallaghan Thanks for showing your link the World Map using World Bank API. In the future can see a map like this being used on Econ Factbook to display data as another page.
@mcallaghan, can you share your code with us in the original repository? Please create a new branch or just tell us where we can find your code. That way we could potential speed up the process of fetching data and place it into a nice and visual tool.
Thanks!