Benefits of releasing historical UKMO observation data

Description & Request Overview

  • The UK Met office (“UKMO”) has an extensive historical database of raw weather observation data. There are currently c.200 UK weather stations used by the UKMO to collect synoptic data (http://www.metoffice.gov.uk/climate/uk/networks/), with an average of c.40 parameters per station including Temperature, Wind speed, Wind direction, Cloud cover (rating 0-9), Solar radiation, Relative humidity in %, Precipitation in millimetres, etc. 
  • In essence, historical weather observation data can be used to analyze and study the past to identify trends and patterns to help forecast weather (i.e. calibrating weather prediction models)  and how weather correlates with other variables (i.e. temperature and crop yield, energy use or road temperatures, rainfall and flooding / flood defences, wind speed and wind farm power generation).
  • We believe this should be made available at reasonable cost for full access to stimulate private enterprise that can provide wider benefits to the economy and society.
  • Currently the UKMO does provide hourly historical weather observations data sets (combination of free and paid for) but these are all limited in some way, either by access, time / date, level of detail or pricing (higher frequency data is recorded but not made available). For example the current free service is limited to a rolling 12 month period only via the UK Met offices Datapoint service, along with less than a quarter of the full list of weather parameters in the full hourly weather report. Longer term historical observation data is available, but is based on monthly averages for a few select parameters.
  • The applications cover a wide range of industries and uses (see 1.2 below), much wider than what we believe can be adequately addressed by one organization, the UKMO. 
  • Some of these applications link with broader government objectives such as renewable energy, climate change, growth of the private sector etc.

Data Release Rationale

  • Private sector enterprises can use the raw data and add value (i.e. through tailoring, interpretation, presentation, collating with other data sources, etc) to create innovative and often highly specialized products and services for specific customer needs.
  • The application of historical weather observational data can be used in many industries including:
    • Transportation i.e.  Route optimisation and winter maintenance for the roads, rail and aviation industries, improving safety and efficient use of resources.
    • Energy i.e. site assessment, operational management and power management for renewable energy and more traditional uses such as load forecasting.
    • Agriculture i.e. choices of crop/mix, land to cultivate, frost protection, animal selection etc
    • Water management: i.e. planning the size and operation of reservoirs
    • Insurance i.e. claims verification and risk assessment to correctly price premiums and incentivise customers to take preventive actions, 
    • Construction i.e. optimal planning of construction activities 
    • Media i.e.  provision of weather information direct to consumers
  • Given the wide range of applications and that new uses for weather data continue to be developed (i.e. Smart grids and energy consumption)  the value to the economy in driving weather related commercial enterprise, risk assessment and resource allocation is significant.
  • By opening up the full suite of raw data collected and archived regularly by the UKMO as a public service, the value of this data could be more fully leveraged by the economy / society as a whole.

Can you identify further areas where this dataset release will create opportunities for innovation and new business?

Can you help identify sectors, businesses and organisations that will benefit from the release of these data?

Benefits Description

  • The primary users of the Historical UKMO observation data would be private enterprise (companies, SMEs or individuals) allowing them to develop and/or improve products and services for the private weather market.
  • The benefits provided to UK society due to increased private sector activity in weather includes:
    • Increased economic activity leading to employment, profits, taxes etc;
    • Improved services provided for other private enterprise / public sector organisations that can drive efficiencies and safety
    • Increased range of services provided by private enterprise and innovation that competition creates – benefiting the consumer and society at large

Benefits analysis

  • Quantifying the benefit of the release of UKMO historical data is difficult due to:
    • Historically, the weather industry in the UK and Europe has been dominated by the national Met offices and therefore the private weather market is still relatively nascent;
    • There is a lack of information on the size and revenues of the private weather market;
    • Historical observation data and its uses only comprise part of the market. 
    • It has a wide range of uses (see 1.2 above for various sector examples)
  • However, there have been estimates calculated for the total private weather market for the US and EU, which can be used as a guide for the potential impact.
  • The total value of the private Weather markets (excluding the large financial weather derivative market) was estimated to be c.$1.4bn in the US and c.$0.5bn in the EU by PRIMET (1), and was broadly supported by other studies (2 and 3).
  • As a % of GDP (4), the US private weather market is c.0.009% of the economy compared to c.0.003% of the EU economy. A key difference between the two markets is that in the US data produced by the National Weather Service (UKMO equivalent) is available for free (plus the additional cost of distribution), whereas the main Met offices in Europe charge / restrict data.
  • For simplicity, assuming that EU weather market has the potential to grow to a similar relative size if restrictions were lifted, then this would increase the EU market to c.$1.6bn.  Again, for simplicity, assuming the UK’s share is in proportion to its share of GDP, this would result in the UK weather market increasing from c.$70m to c.$225m, a $155m increase (c.£100m), providing additional employment, tax revenues, etc. 
  • It is important to note that the market associated with the raw observation data is only part of the market and that the data is already available on a paid for basis, therefore, the impact of this data release would be significantly smaller.
  • None-the-less, the private sector has already proven it is capable of leveraging weather data, with private weather companies having grown significantly as more public weather data has become available.
  • For example, the UKMO’s commercial arm currently represents c.16.6% of total turnover totalling c.£32m and over the last 4 years has grown by c.£3m, a CAGR of c.3.3% (5). This is in comparison to the leading European private weather company having grown revenues over a similar period by a CAGR of c.12.5%, generating c.£28.9m in 2011 (6).

 Input on further case studies or quantifiable evidence to support the release of this dataset would be welcome

Barriers and Requirements for Release

  • Higher quality data at a reasonable cost would allow private enterprise to innovate and drive revenue growth, as well as encourage new entrants to the market by reducing a key barrier to entry.
  • Additional effort should be minimal as the data is already collected and used by the UKMO and its commercial operations. In addition, the UKMO already releases / distributes a significant amount of data of a similar nature.
  • The current commercial operations of the UKMO would not be unfairly affected, as no IP or internal know how needs to be released, but only the raw observation data. 
  • In order to achieve the aims of this request the data should:
    • Contain all the raw data as used and recorded by the UKMO i.e. all time stamps, all meta data descriptions, all weather station parameters and instrument data
    • Data should be formatted and distributed in a usable format
    • For all relevant periods available
    • If charged, it should be the incremental cost of delivery / formatting.
    • There are minimal risks of fraud and privacy.

 Are you aware of any other barriers to this data release?

 Recommendation

  • Further research could be performed to provide more detailed analysis. However, making historical raw observation data available for free (plus reasonable distribution re-work costs) and reducing limitations (frequency / all parameters / meta-data) should be a relatively straight forward process for the UKMO.
  • Therefore, given the impact of opening up weather data in other economies, and the UK so far, we believe this would lead to an increase in private sector activity.
  • On this basis, we believe this should be progressed.

Sources:

  • PRIMET: The Association of Private Meteorological Services: Exploiting Meteorological Data (August 2011).
  • The Private Sector in Meteorology : An update, Bulletin of the American Meteorological Society. David Spiegler (2007)
  • Board on Earth Sciences and Resources (BESR): Fair Weather: Effective Partnerships in Weather and Climate Services (2003)
  • "Nominal 2011 GDP for the world and the European Union (EU).". World Economic Outlook Database, October 2012. International Monetary Fund. 
  • UKMO statutory accounts
  • MeteoGroup – as part of the Press Association statutory accounts
Category: 
Sector: 

Comments

Releasing gridded wind averages as well

Whilst the data seems useful, I have received the following request from a renewable energy researcher for another UKMO data set, gridded averages for wind speed and direction which are described herehttp://www.metoffice.gov.uk/climate/uk/about/methods.html"The data itself doesn't seem to be freely downloadable although they present a comparison with the older and less accurate NOABL data base with summaries herehttp://www.metoffice.gov.uk/renewables/wind-mapIt would be extremely beneficial to our methods and to the small scale industry in General to be able to download the UK gridded map of wind climatology."

So the environmental benefits of this release could be substantially enhanced if such gridded wind averages data was also included

Thanks

 Flag as offensive 

Historic data is required for some uses

 

I was one of people selected for the 2012 Nominet Trust charity ‘Open Data Days', where individuals with knowledge of Open Data spend a day (or, as it turned out, several) helping out a charity with data questions.

I worked with a charity whose primary activity is facilitation the writing and production of a monthly magazine, which is then sold. All aspects of conception, production, and sale are done and run by those with recent experience of homelessness, with paid staff supporting those activities. The organisation does a large amount of good work, however, their ability to get and sustain funding is dependant on the data they show about their activities. Personal stories, qualitative data, is present in abundance; the issue was the quantitative data. 

The financial benefit of such programmes is overwhelming, when the savings to the NHS and Council emergency support services are considered; however, such programmes need to justify their existence on an ongoing basis for funding renewals. 

On entering the office, everyone is asked for their name, and how they are feeling, based on picking a facial expression from a list (on a touch screen). The same questions are then asked as people leave (with time stamps of entry/exit).  This is used as a metric of both time and benefit of involvement. One obvious, substantive flaw in this measure, is that those with recent experience of homelessness are disproportionately affected by the weather, whether through standard of accommodation or means of travel. And the the impact of "getting out of this weather" is wholly absent from the data currently collected, but has dramatic effect it's the use and meaning.

Freely available historic weather data would allow for easy matching, allowing correlation of days when someone was happy to enter, and sad to leave, because of the blizzard or rainstorm outside.  This analysis could simply be included within the tools and software which are shared between NGOs.

Without free and open historic weather data available to all, this is impossible.

Sam Smith

@smithsam

 Flag as offensive 

Data Release

Fully agree with the benefits listed above, there needs to be more in the way of availability of weather data in the United Kingdom. Working in construction, which has been hit heavily by bad weather in recent years, even the most simplistic data such as temperatures and snowfall would be a bonus.

There are only limited private or independent organisations such as https://www.essexweather.com/historical-weather-services.html which provide downtime reports but once again there are limitations on many parameters such as ground temperatures and snowfall depth.

 Flag as offensive 

Climate predictions

It is also highly desirable that climate predictions should be freely available. At present, viewing them involves registering, and they are not available for commercial use.

Surely planning for a range of climate change scenarious is a public good. Accessibility of the predictions would help.

 Flag as offensive