Time to upgrade infrastructure? The SHA1 problem

open-data

#1

With plenty of fanfare, this week, the information security industry has launched public-relations for its software upgrade and consulting services, about the popular SHA1 standard. There are good foundations: see shattered.io, where security researchers announce “first practical SHA-1 collision attack”.

If you is not a programmer, you can understand this “SHA1 problem” as, something like the Millennium bug… It was hyped early, community was properly prepared for replace need: in this case from SHA1 standard to SHA256 standard (or other hash of the SHA2 family). As in the Millennium bug, there are some IT and software industry’s opportunism.

So, there are two distinct and interesting sub-topics to discuss about it:

  • how to (with less impact) upgrade OK projects;
  • the transparency of govern-industry contracts (and costs) to fix the problem.

SHA1 in Open Knowledge projects and infrastucture

Technically, the best strategy I’ve seen, with low impact on databases and interfaces that present the SHA1 code (160-bit long), is the truncation: just change the SHA1 function to SHA256, and truncate the output to 160 bits. I suppose that is totally safe (as comment here). Another way, when you can’t change original checksums but can add more information, is to add more one hash: see this recent discussion about “how to improve long-term duration of standard checksum?”.

SHA1-deprecation and opportunism

The “SHA1 deprecation” was initiated in 2005, was officially deprecated by NIST in 2011, was reinforced in 2015 and finally this week with the shattered.io report (so 2017)…

Between 1995 (the NIST publication of the standard) and 2005 a lot of softwares and services adopted the SHA1 standard without no prevision about how many years the SHA1 be useful. A lapse of 10 years for the most cautious organizations drop SHA1. More ~10 years (you can adopt 2015 or 2017 as reference) for small firms and “less cautious organizations” (a lot of them) change its SHA1 use…

The order of magnitude is a decade (!), and the discover of a collision attack don’t change a lot the so… It is a slow and smooth curve of the changes and costs-of-changes. Not need a sudden change or big costs. So, some suggestions:

  • transparency and technical details are necessary in contracts where the object involves “long-term integrity and authenticity verifications”.
    … Most of “fix SHA1” (upgrade hash) services are simple and low cost… And usually is recommended to preserve old hash codes, only adding a new long-term hashes.

  • comparation of costs: the CPU and human costs, to make a collision-attack (like the reported by shattered.io), can be compared with the value of the object at risk; and the “upgrade investment”.
    … Cost balance is the main factor to show, at each moment, that there are no integrity/authenticity risks, or to justify some access limitation to communications tools.

  • reliability vs security in git repositories (like Github/okfn widely used by Open Knowledge). See this interesting considerations.

  • change from short-term to long-term: some contents published in the past in government gazettes was supposed for short-term duration, but today, with transparency pression, we see it as long-term. In third world is also usual that a software vendor say to government that a solution is for long-term preservation, when isn’t really.


#2

Hi @ppkrauss

Are you making a suggestion for OKI infrastructure? I’m not sure based on your post.


#3

Hi @pwalsh, it is a two-in-one topic, for technical and political discussions :slight_smile:

  1. Open Knowledge and OK-Chapters have infrastructre do be checked (perhaps the best discussion here is to obtain a consensual check-list)… in chapters like OK-Brazil, there are also a lot of software projects, that need some attention to review SHA1 use in it.

  2. The general consequences of SHA1 news of this week, mainly the estimated costs:

    2.1. General transparency/spending of local-governments in that kind of problem (above described as Y2K fanfare)… In Brazil we have a lot of examples of “opportunistic corruption”, in company-government service contracts.

    2.2. The problem of digital preservation in a perspective os decades (authentication certificates by hashes) that needs immutable records, as in the case of official gazettes
    The pression of security industry is to “refresh preservation records” (that have cost and create audit instabilities) at each event of hash-deprecation. Perhaps we can endorse good practices that avoid this kind of “refreh demand” in digital preservation.