Hi everyone, I am working on an economics website at econfactbook.org
The goal of Econ Factbook is to illustrate the world economy with statistics. I am looking for a programmer to help me make a bot that can automatically update the website. This bot would retrieve data from a variety of sources. Please let me know if you might be interested.
I am open to ideas on how to improve the website. If you have any ideas, please post your suggestions.
Hi @alexpeek1,
Thanks for this link. Very interesting indeed.
Is it possible to retrieve specific sets of data to place them on a dataset online, somewhere? In that I would be interesting. Plus, keeping your site is important, as it is a graphical alternative to observe the data.
I’m not master in programming, but I don’t mind getting my hands dirty to find a way to create a script for fetching the data and do the website. I also know a couple of designers that could help us taking care of design.
Let me know if any of this sounds interesting.
thanks Gustavo,
I would be interested in working with you and I would also be interested to talk to some designers.
I want the website to be automatically updated with the most recent statistics. Right now it is just a collection of manually updated txt files. Much of the statistics are from excel spreadsheets on the internet, other statistics are directly from webpages, some statistics are from PDF’s. To be able to apply a script to retrieve the data, what would the website need? It would be nice if the bot could automatically know when new data has been released also. I dont know how to set up an API.
Indeed, very interesting. I’m also not an expert in programming, but I’m inputting a link “library” I have worked on of open data readable in GIS and useful for humanitarian crisis response into a map. Some friends are helping and we can discuss more about it if you want. Maybe they can send you a tutotial?
More soon
Bruna.
Thank you for your interest, @alexpeek1 !
My plan was to convert all the data into csv files and post them online, both in CKAN (English) and Central de Dados (Portuguese). The problem with a few PDFs is that its document structure is regularly changing, so there’s no possible way to automate things.
If some sources are spreadsheet based, then it is easier to work with something to build the final results.
I’m no expert, but I’d say you will need Javascript or some of those tools that automatically do things for you. Central de Dados is a good example in that regard.
About the designer things I said, I didn’t reach them, but I’m sure they will contribute with ease.
Perhaps @Bruna_Santos can also help taking care of putting the data online, specially when she has experience doing so!
@alexpeek1: I would like to start working on this project soon. Can you tell me which platform do you prefer? Email, Skype, this forum (would be nice), etc?
@Bruna_Santos, do you want to contribute to this too, or not so much at the moment?
@gsilvapt thanks for your interest to get involved. I think it would be good to talk on the forum or by email. Should I create a new domain for the website or can it be done on econfactbook.org? I have not set up an API or anything so I am not really sure how to do this. I agree that the data in csv files should be posted online somewhere, but I am not sure how this can be linked to my server/website. Is there something I should do on my end that would allow you to link statistics directly to econfactbook.org? All of my best sources of economic data are already posted on econfactbook in the reference section.
@Bruna_Santos Thanks for the support, I would be interested to see a tutorial. sounds like a cool project. If you are interested to get involved with econfactbook please let me know. one of my goals is to apply the format to every city in the world.
Okay, we can discuss things here.
I’m more used with R, but I can definitely learn how to work with Python. Actually, this might be an ideal project to get my hands into.
My idea is to break this project into two:
For part two, if you tell me which language are you using, I will reaching out to those people I know and hopefully we’ll find a couple of volunteers to work with you.
@gsilvapt That sounds like a good plan.
Part 1: We can use R programming since you are more experienced with R programming. Where should the repository be located? And how do we get data from the repository onto econfactbook.org?
Part 2: I am not using JavaScript yet but I think that is a good idea. I have only used basic html and Google fonts. I am not using any programming language yet but I am open to any suggestions. JavaScript should be good for front end design.
Good luck reaching out to people, hopefully we find a person to help make the website look nice.
We will see about that. I believe we will find a way to have this on openeconomics.net, and then we can spare you the costs of the domain and server. Nevertheless, I don’t want you to bury yourself with costs of open source projects - that does not make sense, and that is why OKFN exists.
I’m more used to, not a pro 
R is interesting, but perhaps Python suits better for this case and it is also “universal” and others may help as well. Using R may restrict contributions because not all programmers (common in open source projects) know R. Also, I’ve been trying to find a project that suita my Python skills (basically none) to do it. In this case, I believe we’ll need to have some automated tasks with all links, fetch and create separate csv files to post it to a database, which can also be automated if we use Python. R does not allow that. By the way, how comfortable are you with Python?
As for the project hosting, I will take care of creating Open Economics as an organization on GitHub. If it already exists, I’ll try to get access to it and then we will work from there. How does that sound?
Great to hear! I already reached out and I’m hoping to find someone too.
@gsilvapt I like your idea to find a way to have this on openeconomics.net. That seems like a good idea to me.
I think Python would be best. Most people have told me that Python would probably be best for this kind of task. Especially if Python will allow for greater automation. I have a little experience with Python. The only thing I used Python for was to read excel spreadsheets to produce text html document scripts I could use for the website. That is the extent of my Python skills.
Let me know when you have created the Open Economics organization on GitHub so I can see what you are talking about exactly. That would be the place to upload statistics?
Thanks for your interest so far and answering my questions.
Okay, Python will be then!
I’ve created Open Economics on GitHub, you can find it here. However, I believe it is better if you give me your email address to add you to the organization (just to keep things organized and not too messy).
There is nothing for you to thank me. In fact, I have to thank you for being okay with using your main idea to improve Open Economics 
Good to hear. my email address is alexpeek1@gmail.com
Invitation sent. You can block your email for security reasons.
Ok cool thanks for the invitation. Is there anything at this point you recommend that I start working on?
If you want to post everything about the website online, just so that other people can edit it if they want to, it’s fine.
Other than that, I’m still waiting for OKFN’s response in regard to the website. I still have a backup plan, but I truly don’t want to go there. Also, I’m busy creating very basic tutorials and information for people to know how to create and publish datasets. After that, I’ll move on to something else
If you want to, you are free to do anything you’d like to 
Should I just make a new file on the datapackage-validator page? I am a little confused on where I should post things about the website.
I want to create a new repository for this. Maybe name it econ-factbook. Inside, we can either separate things into two different folders (say, website + database) and work from there, or test some other structure.
I never tested the permissions, so this might actually be the right time to test if contributors can create a new repository
Ok that sounds good. I think two different folders would work.
Good! Can you create that repo for me then?