This is a really important topic and something we covered a little while back with the ODI by looking at inherent quality issues in Companies House, Land Registry and NHS data. In essence, publishers need more support with resources, standards & tools to get better data out there.
Strong collection standards need to be enforced (e.g. use only valid Country names when registering a business or require GP surgeries to give valid phone numbers to NHS Choices) and tools are needed to automate the extraction from source systems into the 'open' standard format (e.g. local authority spending data).
It's not impossible but it needs effort & some funding. We also need a single set of definitions, standards and the involvement of the community to improve data - some form of open data GitHub.
My main concern is - who should co-ordinate this? As an earlier post mentions, there are guides from W3C, ODI, OKI and probably others, but are they all the same? How do they differ from UK Local Government Association guides? A single entity needs to lead it.