Local Spending Data Guidance
The document below (and attached to this blog post) contains draft guidance for Local Authorities on how to comply with the Prime Minister’s call to publish each financial transaction over £500 from January 2011. As the guidance is in draft format, we would welcome any comments, suggestions or questions you may have. You can either post your comments below or contact the Local Data Panel directly. We look forward to hearing your feedback to the draft guidance.
Local Spending Data guidance
This guidance has been produced by the Local Public Data Panel, and builds on the draft guidance published in June this year by the panel, on comments made on that guidance, the Treasury guidance to central government departments on releasing their spending data, and experience from publishing the data, and, crucially, using and analysing it.
It is designed to give councils clear standards for publishing their spending data, setting both minimums that should be achievable by all councils, and gold standards for those councils with the ability to publish to a richer standard.
Because this is a rapidly evolving area, the guidance is expected to be revised in 6 months’ time based on experience, feedback, and at that point the minimum standards will be increased (as will probably the full, gold-standard ones). All councils should publish to the minimum standards within 6 months of the publication of the guidance.
Although this guidance specifically relates to the publishing by local authorities of spending data over £500, there is much to be gained by other local public bodies (health, police, waste authorities) publishing to the same standards where possible.
1. Overriding principles
The purpose of requiring councils to publish their spending data is to enable greater transparency, greater engagement and greater efficiency by allowing inspection by residents, and reuse of the data in innovative applications, mashups, visualisations and analysis.
Given this it is essential that councils publish their spending data (and other data too) with a licence and a format that allows reuse, and on the basis that data should be published unless there is an overriding reason not to (principally to protect vulnerable people and privacy of ordinary citizens). This means that councils should be be asking, what’s the most they can publish, not what’s the least they have to do.
The focus of the guidance is on how, pragmatically, to make the data available quickly rather than seeking to achieve full alignment across every entity. This is consistent with the Sir Tim Berners-Lee’s paper, ‘Putting Government Data Online’. Publishing raw data quickly is an immediate priority, but there are significant benefits to be gained by councils publishing structured, regularly updated data using open standards, and this guidance helps enable this.
There is, however, no reason why, if they have the resources, authorities should not publish in more ways, or in greater detail than in this guidance. For example, some may start publishing invoice data, others publishing the information as linked data (in which the data is semantically described using URIs), and we expect to include the successful practices learnt from such experiments in future versions of this guidance.
2. Licence, Timing & Contacts
Data should be published in a timely manner and with a licence that allows open reuse, including commercially. We recommend the new data.gov.uk licence.
There should be a nominated contact person, listed on the page linking to the data files, to oversee the publishing of the data, and this person should also be responsible for answering queries to do with the data, and requests for further information. This will not only increase the skill levels within the council and improve relationships with reusers, but also reduce the necessity of going down formal routes (e.g. Freedom of Information requests).
Data should be published monthly, as soon as possible after month end but no later than 30 days after month end. This data is to be published in individual monthly files, and there should be a single web page which contains links to the individual files. Files should be named in a consistent manner with the date included in the file name, and each file name should be unique. You should avoid spaces or characters other than a-z, 0-9, _ and -.
Authorities may wish to publish the data for longer reporting periods e.g. quarterly, year to date, etc. There is no reason why they should not do this, but this should be in addition to the individual monthly files.
The Secretary of State for Communities and Local Government has committed local authorities to publish items of spending over £500 by January 2011. We recommend that where possible data is provided from 1 April 2010. This will add value to the user and demonstrate commitment to complete transparency.
When errors are discovered, or files are changed for other reasons, rather than ‘silently’ changing the file it is recommended to publish a revised and differently named version (together with the original files), e.g. 02_2011_v2, 02_2011_v1.
The spending data should be published as part of a wider open-data initiative, and there should be a dedicated open-data page/section on the council’s website (the recommended URL for this section is www.yourcouncil.gov.uk/opendata), as well as listing the files at data.gov.uk.
3. File Format
The files are to be published in CSV file format. Microsoft Excel files should be converted to CSV. The CSV file must have precisely one header line with field names exactly as in the example file supplied. Values must be separated by a comma character, with a new line between separating transactions. Text values that contain a comma must have a ‘"’ character at the start and end of the value. This format is the behaviour of Excel when using the “Save As” function selecting CSV as the file type. There should be no comment lines, no ‘total’ lines and no blank lines.
Due to limitations of Microsoft Excel, it is unable to deal with non-Western characters (like Chinese). In cases where you have such payments, an alternative route to creating the CSV files in Unicode format will need to be found. In the interim, councils needing to publish data including non-Western scripts should provide Excel files. Note that ‘£’ signs can also cause problems for the same reason and thus should not be included in the file.
Authorities may wish to publish the data in additional formats as well as the CSV files (e.g. linked data, XML, or PDFs for casual browsers). There is no reason why they should not do this, but this is not a substitute for the CSV files, and PDFs in particular do not allow their contents to be reused. Authorities must take full responsibility for the content of their files.
4. Publication requirements and process
Consistent with producing raw data quickly, the published data should reflect how each individual item was originally recorded in financial systems, including all related information (e.g. supplier, invoice, procurement details).
It’s worth acknowledging that some authorities provide shared service operations, and these should provide the data in the required format to the authority to which they provide services. The entity that owns the data is responsible for its publication. Authorities will need to put arrangements in place with their shared service providers to obtain the necessary data.
All payments to suppliers for goods and services and grants over £500 (including VAT) should be published, including all individual invoices, grant payments, payments to other public bodies, expense payments or other such transactions. Credit notes should also be published.
Salary payments to staff should not be included, nor should compensation payments made to individuals. Publication of salaries to senior staff will be covered under a separate guidance note. Certain other transactions may be redacted or be exempt from publication. These are covered in more detail later.
There are limited exceptions surrounding personal information that falls under the Data Protection Act, mainly when dealing with payments to vulnerable people (e.g. payments to foster parents). Payments to any redacted data entry should be replaced with the word ‘REDACTED PERSONAL DATA’ or ‘REDACTED COMMERCIAL CONFIDENTIALITY’.
Note that payments to individual consultants, or sole-traders or other supplying individuals should not normally be redacted, and in all cases supplier ids should be published. In the case that this may identify a vulnerable individual, because, for example, the payments are made to someone who happens to be a sole-trader supplying the authority, we would expect there to be different supplier records for the payments relating to the vulnerable person, and if not a new one should be created. There are no circumstances in local authorities where payment amounts and the record of the payment itself should be redacted as a whole.
The Freedom of Information Act should be used as a frame of reference when making these judgements. The credibility of transparency reporting would be seriously undermined if expenditures were subsequently published under FOI. You should bear this in mind when forming your judgements. It should also be noted that the data to be published, will, in most instances not disclose price, and therefore is unlikely to be commercially sensitive. Together with recent rulings by the Information Commissioner (favouring the right to information over ‘Commercial Confidentiality’) this makes it highly unlikely that any local authority transactions should be redacted for this reason.
Where payments relate to several different categories or invoices these should be broken down into their constituent parts.
The minimum threshold of £500 is a requirement, but there is nothing to stop local authorities publishing all spending, and as systems for redacting information relating to the identities of vulnerable people improve it may well be easier not to have any threshold, and this will certainly aid in tying individual transactions to the aggregated figures in the authority’s accounts.
Councils have also been told they are expected to start publishing new contract and procurement information from Jan 2011, and so all information linking procurement and accounts systems should be published, allowing spending to be linked to contracts and tenders. Procurement systems will also typically have much more information about suppliers (e.g. Company Number) than accounts systems and this will make it much easier for users of the data to identify them.
Publishing the supplier list (matching to the supplier ID) and also the authority’s Chart of Accounts will also help users make sense of the data and this should be published.
Because the publication of the data becomes a public view of an authority’s accounts there will be benefit to publishing corrections/journals.
The ‘meta-data’ about the accounts system should be published (including software on which the accounts and related systems run and procedures/associated programs used). This should be published as an accompanying plain text file, or on the web page linking to the data. This meta-data should also include any departures from the standards listed in this guidance.
5. Data content
The content of the published data must match the format set out in Appendix i, below. The data must follow the sequence of columns prescribed, and any formats or presentation conventions as set out in the general commentaries below. An additional fields that an authority wishes to published should conform to the general specifications (e.g. quoting values containing commas), and should be added after the fields listed here.
- This identifies the local authority to which the data relates, and means that the file is self-describing (i.e. all the information needed is within the file). It should be the URI that represents the local authority at statistics.data.gov.uk. In the case of UK local authorities, this is of the form of http://statistics.data.gov.uk/id/local-authority/XXXX (where XXXX is the ONS SNAC id for the authority). In the case of the GLA (which doesn’t have a SNAC id) the following URI should be used: http://data.ordnancesurvey.co.uk/id/7000000000041441 . You should also add the name of the authority in the Body Name field to aid readability.
- The date when the transaction took place. The date should ideally be the payment date as recorded in your purchase or general ledger. This is not the date when the supplier receives the funds. The date must be in the UK format DD/MM/YYYY. However, if it is not possible to source the data using the payment date, then you should first look at the possibility of using the date the invoice was input to the system, otherwise you should use the invoice date.
Use of date must ensure that transactions are only disclosed once, and must be consistently applied. If you are not using the payment date, you must note on the meta data the date you are using and why it is not possible to use payment date. You must also advise what parameters will be used for your disclosures (i.e. how is data being selected for inclusion, so that transactions are only disclosed once).
- Transaction number
- This should be system transaction number for the payment held by the individual department. This may be useful to the authority, should there be any follow up questions about an individual transaction.
- Amount in sterling
- The amount disclosed should be the amount recorded on the finance system for each individual transaction. To identify the transactions to be disclosed you should identify all transactions above the reporting threshold with reference to the total amount payable inclusive of VAT. Transactions should also be selected gross of income. All amounts published must be in sterling and in pounds and pence. If a single invoice has been coded to multiple expense types, and/or expense areas the value shown should be the amount paid against each individual expense types/expense area combination, even where each entry at this level may be less than £500.
Where possible, recoverable VAT should be excluded from the amount disclosed (i.e. shown on a net basis). Where the recoverable VAT can be excluded from the disclosure it does not need to be separately disclosed, and can be omitted completely. Where removing recoverable VAT reduces the value of the transaction below £500, the transaction should still be disclosed. However, if the source of the data being used cannot separate out recoverable VAT then the gross amount should be used instead. You should state on the web page on which the data is published (or in the accompanying meta-data text file) if you are unable to publish amounts net of VAT. Where VAT is irrecoverable the total transaction amount should be disclosed.
- Supplier details
- In order to properly understand where and with whom public money is being spent, it is important to be able to identify the supplier. This is often not possible from the name alone (which may not be the formal name of the company or organisation), and ideally publication would include the Companies House number (or equivalent in the case of foreign companies), charity registration or other unique identifier.
In practice this information often isn’t held by the accounts system, and because of that the guidance is that all authorities should publish the supplier name and supplier id (to distinguish between different suppliers with similar names and to match the same supplier when the name changes or is redacted).
In addition, where the information is held, authorities should publish the supplier VAT number (although there is no exact match between VAT number and organisation, experience has shown this is a very successful way of identifying suppliers), as well as publishing the supplier list separately as open data.
The name of the supplier should be as per your vendor record. If the payment you make to the vendor (e.g. a solicitor) is to enable them to make a payment to a third party, you need only disclose the name of the original vendor not the end beneficiary. If the same supplier appears more than once under different vendor records, this is how they will appear in the published record. Payments to individual suppliers should not need to be aggregated, and each individual transaction must be shown separately. The supplier name should appear in full.
- Expense area
- This is the part of the entity that has spent the money. It should represent a meaningful part of your organisation structure or activity. This might be at directorate level, or another appropriate part of your organisation structure, possibly drawn from your cost centre structure. To aid transparency, it would be helpful if the structure you select were capable of being mapped onto your published organisation structure. Where a consolidated invoice is received, you may reflect the original coding of the transaction, rather than the subsequent reallocation to different parts of the organisation. However, you must still designate an appropriate expense area.
- Service categorization
- In order for spending to be comparable between periods and between local authorities it is important that it is clearly and consistently categorized. Depending on the accounts system this may be easy or quite difficult. There are two candidates for categorization – CIPFA’s BVACOP classification and the Proclass procurement classification system (which is widely used in local authority procurement systems).
It is worth noting that these provide different classifications – BVACOP is an accounts classification, and Proclass procurement classification. It’s also important to say that while Proclass is published under an open licence, at time of writing BVACOP is not, and is therefore not suitable for use when publishing open data, although there has been talk that this will change.
The recommendation therefore is that local authorities where possible should publish whichever classification they can (subject to BVACOP being published under a licence compatible with the data.gov.uk licence), and ideally both.
All authorities should however publish a consistent Expense Type (i.e. the Service description), together with the ID identifying the Service (possibly the nominal code on the chart of accounts).
- Original draft guidance and comments http://data.gov.uk/blog/publishing-itemised-local-authority-expenditure-advice-comment
- Treasury Guidance for central government http://www.hm-treasury.gov.uk/d/transparency_spend_over25100910.pdf[PDF]
- Tim Berners-Lee’s paper, ‘Putting Government Data Online’ http://www.w3.org/DesignIssues/GovData.html
- Balancing Public Interest and Commercial Issues http://data.gov.uk/blog/payments-over-%C2%A3500-%E2%80%93-balancing-public-interest-and-commercial-issues
- CLG press release re local authority spending http://www.communities.gov.uk/news/corporate/1606882
Appendix i: CSV file format
|Column||Field Name||What Is Required||Reason For Inclusion||Additional Information||Inclusion status|
|1||Body||The statistics.data,gov.uk URI that represents the authority, e.g. Lichfield District Council is http://statistics.data.gov.uk/id/local-authority/41UD||To allow the file to be self-describing||Using a URI to identify the body means that it is uniquely identifiable to data users. The GLA should use the URI http://data.ordnancesurvey.co.uk/id/7000000000041441||Mandatory|
|2||Body name||Name of the authority||Aids readability for casual reading||Mandatory|
|3||Date||The payment date as recorded in departments’ purchase or general ledger.||To identify the date that cost the transaction took place.||If it is not possible to source data using the payment date invoice input date or invoice date may be used The UK date format (DD/MM/YYYY) should be used. Leading zeros should be used where necessary so that the string is precisely 10 characters (e.g. 01/09/2010)||Mandatory|
|4||Transaction number||A unique reference number for each individual expenditure transaction||To act as a reference number when dealing with enquiries or FOI requests.||The transaction number used in departments’ own systems may be used.||Mandatory|
|5||Invoice Number||The reference number for the invoice if it is recorded on the accounts system||To allow matching to invoices (one payment may be for several invoices, and one invoice may be settled by several payments)||Some councils have been discussing ultimately publishing invoices, and inclusion of this fields allows for an easy transition to this, should they do this.||Desirable|
|6||Amount||The actual value of the transaction||To identify the full cost of the transaction.||Amounts should be in sterling and inclusive of irrecoverable VAT. Values should be in pounds and pence. Each entry should include a decimal point and exactly two digits for pence. Pound or other currency signs must not be included. Income or other negative spend (eg corrections) should be show with a leading minus sign. Leading zeros should not be used. Commas should not be used to separate thousands of pounds. So, for instance, a payment of £25,123 should be shown as 25123.00 and a credit of £26,123.45 should be shown as -26123.45.
If possible recoverable VAT should not be included and does not need to be shown separately.
Expenditure needs to be published gross of any income.
Where an invoice crosses multiple expense codes, the separate payment lines should be published even where they are individually worth less than £25,000.
The amount should represent expenditure for an individual transaction and should not be cumulative.
|7||Supplier Name||The full name of the supplier||To identify the recipient of the spend||The name of the supplier named on departments’ own vendor record can be used. Where the same supplier has been recorded using different naming conventions, there is no need to aggregate. However these multiple versions will appear in the published record.||Mandatory|
|8||Supplier ID||The internal ID of the supplier in the accounts system||To allow suppliers with same name to be distinguished form each other and allow matching of the same supplier even if the name is changed||Mandatory|
|9||VAT Registration Number||The supplier’s VAT registration number||To identify and aggregate supplier information even if the supplier descriptions are not consistent||Publishing the VAT number makes it much easier to match the supplier to external data, and thus makes comparisons across authorities much easier||Desirable|
|10||Expense Area||The Department or Directorate responsible for spending the money||To identify the area within the authority that has spent the money.
The description needs to be meaningful and may possibly relate to the department’s organisational structure. This may be at directorate level or relate to the cost centre structure.
|11||Expense Type||The description of the type of expenditure||To identify the general nature of the spend||The description of expenditure used against account codes held on departments’ own finance systems should be suitable for this purpose e.g. consultancy, stationery.
Descriptions should make sense and be suitable for the public domain.
|12||Expense Code||The internal account code which represents the Expense Type||To allow expense types to be matched even when the wording of the description is changed||Desirable|
|13||BVACOP||The BVACOP code for the transaction||To allow comparison of spending by finance category between periods and authorities||Desirable|
|14||Proclass||The Proclass code for the transaction||To allow comparison of spending by finance category between periods and authorities, and to aid in connection to procurement data||Desirable|
|15||Extended Description||A description in words for the details of the transaction||This might be stored as a commentary in the accounts system, and inclusion will help explain the payment||Desirable|
Appendix ii: examples of data to be included
The table below gives specific examples of transactions that should be included in publication.
|No.||Examples of transactions that should be published||Reason|
|1||Payments to other government and public sector bodies||All transactions whether with other public or private sector bodies should be included|
|2||Payments to government or other third party service providers||All transactions should be included|
|3||Payments to sole traders||Business rather than personal expenditure|
|4||Payments for secondees||Payment for service rather than personal or pay bill expenditure. However, if a secondee’s pay would become transparent, this should be redacted.|
|5||Travel and subsistence claims|
|6||Service charge element of pension contributions|
|7||Ex-gratia payments above contract price||The full payment cost is required|
|8||Credit notes||Needed to ensure correct transaction values have been recorded|
|9||Policy lending (other than to individuals, or funds management)||Regarded as spend|
|10||Gifts||Publishable under FOI|
|11||Rent and business rates||Standard expenditure costs|
Appendix iii: examples of data to be excluded/redacted
The main principles are expected to follow the exemptions provided by the Freedom of Information Act. Key redactions will relate to data that is protected under the Data Protection Act, particularly relating to children and vulnerable persons. Please note this should be read in conjunction with the examples of data to be included, above.
The table below gives examples of the types of transactions that may be redacted or excluded from publication.
|No.||Examples of transactions that may be excluded from publication||Reason||Redacted or Excluded|
|1||Salary payments to staff (including bonuses)||Personal information protected by the Data Protection Act||Excluded|
|2||Pension contributions (excluding service charge) and National Insurance Contributions||Excluded|
|4||Payments to individuals from legal process - compensation payments, legal settlements, fraud payments||Redacted|
|5||Competition prizes – where a normal part of operations||Redacted|
|6||Settlements made with companies as an arbitration which is conditional on confidentiality||Commercial-in-confidence – exempt under FOI||Redacted|
|7||Potential betrayal of a commercial confidence, or prejudice to a legitimate commercial interest||Very rare and will need to be justified||Redacted|
|8||Transactions relating to the financing or underwriting of debt e.g. purchase of credit default swaps||Outside the definition of expenditure for this purpose||Excluded|
|9||Provisions or promises to pay not yet realised||Excluded|