Can you add code descriptions to a data package?

danfowler · September 6, 2016, 3:15pm

Hi @Stephen this is a great question. It was previously raised on the forum here and moved to the following GitHub issue on the specification repo:

github.com/frictionlessdata/specs

Supporting optionally ordered factors/enums

opened 06:02PM - 17 Dec 14 UTC

houshuang

Table Schema

In both R and Julia, there is a concept of factors or enums as an alternative to… character strings. The idea is to specify a fixed amount of possible values, and possibly an ordering of these. An example could be a questionnaire, which might have a field with Male/Female, Student/Working/Unemployed/Retired, or a likert-style question with Not at all/Little/Neutral/Quite a bit/Very much. The two main advantages are validation (you can ensure that there are no other values than the ones enumerated (plus NA)), and storage capacity. (Internally, these are often implemented through storing ints and a lookup table, compare storing 65,000 likert-style questions above as characters, with storing 65,000 ints). The ability to order the levels (for example Not at all > Little > Neutral > Quite a bit > Very much) means that the levels will always be presented in the same order when graphing or showing summary statistics, rather than alphabetically or by order of first appearance. I typically spend quite a bit of time cleaning up incoming questionnaire data by converting them to ordered factors, and would like to be able to preserve this information when sharing the datasets using dataprotocols. In terms of implementation, there are two options. The first would be to keep the actual data exactly as it is, but note in the metadata that it is a factor with certain levels, and an optional ordering. This would presumably make the file "backwards-compatible", and still enable anyone to simply import the CSV without bothering with the metadata. We could still use the metadata to validate the CSV. An alternative which would in many cases drastically reduce the size of the CSV file, is to store ints in the CSV file, and provide a lookup-table in the metadata (1=Not at all, 2=Somewhat, etc). It's possible the difference between the two approaches becomes very small once the files are compressed, so perhaps that makes this less backwards-compatible approach less important. Discussion and support for this suggestion here: http://discuss.okfn.org/t/something-like-rs-ordered-factors-or-enums-as-column-type/122/4

Topic		Replies	Views
DataPackage for GTFS data Open Transport	1	1370	September 7, 2016
New post on Labs blog: Using Data Packages with R Frictionless Data	1	2208	July 14, 2016
Integrating with a website Frictionless Data	5	1087	February 25, 2016
Geo Data Package Frictionless Data	42	5317	March 1, 2018
Tutorial for handcrafting a Table Schema Frictionless Data	6	1365	June 15, 2017

Can you add code descriptions to a data package?

Related topics