Data packages for bulk downloads

Today we've adding a "bulk download" feature that works with the majority of datasets on data.gov.uk. This allows you to download all the data files of a dataset using one click. The data comes in a zip file, with an description of the data (metadata) included - known as a "data package".

Data Package ZIP

204 downloads or get them all in one Data Package ZIP

Whilst the average dataset has about 5 downloads, plenty of the more substantial ones have many more downloads than that. Often a dataset is helpfully split into several files because many users will just want one particular geographic area, or maybe it's because updates are put into a new file for each time period. But many others users will appreciate getting it all in one bulk download.

Note that many datasets in data.gov.uk do not have direct download links, so it is not appropriate to provide a zip in these cases. This can be because they only provide API access (e.g. SPARQL or WFS), but for a good number it is because despite our efforts to clamp down on it, there are still publishers that have only supplied data.gov.uk with links to web pages about the data, rather than actual download links. In these cases we make it clear that the ZIP is not appropriate (example).

The metadata, which contains a description of the data and its files, is supplied inside the zip in two formats - a human-readable HTML file and a machine-readable JSON file, known as a 'Data Package descriptor'. (This JSON is also available directly from the API panel on data.gov.uk dataset pages.)

Data package metadata

The zip is easy for most people to use, but advanced data wranglers can get additional benefits by working data packages using command-line tools like 'dpm', providing push/pull features similar to the version control system 'git'.

Many in the open data community have talked about the importance of supplying metadata together with the data. And this is embodied in the lightweight Data Package standard developed by Rufus Pollock and others in a move towards 'frictionless data'. Whilst we don't yet provide table schema information in the data.gov.uk data packages, we aim to start adding that in the future.