Yves Buhler, director of the Technical and Scientific Support department (TSS) at EUMETSAT, shares details of the organization’s new data services, which sit at the heart of its long-term big data strategy.
What new services has EUMETSAT launched recently?
We have a number of services that are currently pre-operational and will become fully operational in June 2021. First, Data Store provides a single point of access to the growing catalog of EUMETSAT data.
Eventually, this service will offer the full data catalog, including near-real-time and historical data, as well as climate data records. Second, the Data Tailor service will allow users to customize EUMETSAT data to their needs by applying, for example, format conversions, region of interest subsettings and many other options. Third, the EUMETView data visualization service will allow users to create visualizations, maps and animations using EUMETSAT data. The evolution of weather phenomena can be seen by creating animations through EUMETView. EUMETView is a genuine web map service on EUMETSAT data.
In addition to these new data services, we are regularly enhancing our portfolio of data by adding new data products derived from our own satellites, and use our international cooperation with other satellite operators worldwide to offer our users a complete set of meteorological data for a variety of different applications.
This image shows eight separate events saved for later use via the EUMETView My Views capability. Top row, from left: Cyclone Eloise on January 18, 2021, the day before it made landfall in Madagascar, comprising the multisensor precipitation estimate and natural color imagery from Meteosat Second Generation (MSG) Indian Ocean service; Medicane Ianos, pictured on September 16, 2020, killed four people and caused flooding and severe agricultural damage in Greece. Layers are similar to those used in the first panel; iceberg A68A as seen on November 5, 2020, using top-of-atmosphere, natural color data from Copernicus Sentinel-3’s Ocean and Land Colour Instrument (OLCI); the Alboran Gyre on October 16, 2020, shows concentrations of chlorophyll-a as captured by OLCI. Bottom row, from left: global Metop sea surface temperature data from EUMETSAT’s Ocean and Sea Ice Satellite Applications Facility (OSI-SAF) (with Sentinel-3 OLCI natural color data), capturing western boundary current variability through a solitary meander in the Agulhas Current off the eastern tip of South Africa on January 30, 2021; OLCI true color image of the Caspian Sea on August 30, 2020, shows the turbidity of the northern part of the water body; the pink plume shown on this image from February 5, 2021, shows dust from the Sahara spreading into Europe on this false color MSG RGB imagery; the upwelling of cold water on the western coastline of Chile comprises layers of sea surface temperature and ocean color data from Copernicus Sentinel-3
What do you hope to achieve with these new services?
The expectations of the users of EUMETSAT have fundamentally changed with the emergence of new technologies in the big data era. Users today expect a variety of flexible services using cloud technologies. Empowering the user is a key aspect here.
With these new services we will offer access to vastly more data as new satellite systems come on stream. The EUMETSAT archive contained 6PB of data at the end of 2020. That will increase to about 50PB by 2025. As the volume of data increases, it is important for us to ensure that its delivery remains easy and reliable for our users.
Copernicus Sentinel-6 satellite data (see Monitoring the global ocean, right) will start to become operationally available from mid-year and will be fully operational by the end of the year. The upcoming launches of the next generations of our geostationary and polar-orbiting satellites, starting in 2023, will also contribute additional data sets to make weather and climate predictions and modeling more accurate and pertinent.
Not only will the quantity of data increase, but the quality of the data from these new instruments will be among the best in the world. Hopefully, enhanced by state-of-the-art scientific processing, the EUMETSAT new data services will become a worldwide reference for meteorological data and derived products.
Finally, EUMETSAT data will become available to more users in a variety of formats that will encourage a broader range of uses. EUMETView will allow for easier and more pertinent data visualization, and Data Tailor will allow users to select the format of choice. Another aim is to increase the number of people using the data, whether for applications in weather-sensitive sectors of the economy or for scientists and researchers.
Allowing for easy access and processing on the cloud ensures that all users, even those with less powerful computers, can use the EUMETSAT data and generate their value-added products from the available data sets. Clearly this greater flexibility has the purpose of maximizing the usage and user impact of our data and products for our users.
These new data services will also strengthen the participation of EUMETSAT in wider European and international projects, such as the development of WEkEO as part of the Copernicus program, or of the European Weather Cloud, in the context of the European Meteorological Infrastructure. The new data services are also an important support, both technologically and scientifically, to the EU’s Digital Agenda and the Green Deal. In that context, EUMETSAT will be a core partner for the Destination Earth project launched by the European Commission, providing the knowledge and infrastructure to manage the immense quantity of data necessary to create digital twins of the Earth.
Users can easily access the Meteosat Second Generation data used to construct their views through links to the EUMETSAT Data Store service
What information is provided in the new data services?
The data from EUMETSAT’s meteorological satellite fleet and the Copernicus spacecraft it operates on behalf of the European Commission are available through these new services. This includes data from the geostationary satellites Meteosat-8, positioned over the Indian Ocean, and Meteosat-9, 10 and 11 with their views over Africa and Europe, as well as from the three polar-orbiting Metops, A, B and C.
In addition, data from the Copernicus spacecraft operated by EUMETSAT, Sentinel-3A and 3B, is available, as is data from the ocean altimetry spacecraft Jason-3. Data from the follow-on mission to Jason-3, Copernicus Sentinel-6, will start to become available from mid-year. The data from EUMETSAT’s unique archive stretching back more than 40 years is also progressively being made available through the Data Store.
Why are these services being launched now?
The new services are elements of a long-term big data strategy at EUMETSAT. We deliver cost-efficient, operational services to our member states 24/7, on a high-availability basis. To do so, we need to use the latest technology. With the rapid development of the cloud and internet-based solutions, we also have to continuously adapt.
There are three overarching directions of our strategy. First, to provide a vast collection of data from our satellites and also from third-party satellite operators with whom we have cooperation agreements. This provides the diversity. Second, to establish a high quality standard, both for the accuracy of the measurements and for state-of-the-art processing on the ground to derive the best products from our satellites. We aim to remain world leading in this respect.
Third, to build up an infrastructure and services that make this access to EUMETSAT data quick and easy to use through fully integrated data services. Anyone with internet access will have access to EUMETSAT’s data.
The new services will benefit existing users in the meteorological, climate and research communities. EUMETSAT has more than 8,000 users of our data, including almost 3,000 private users. Our open-view service attracts an average of more than 54,000 unique users per month. The new services can also be beneficial to other, new users, such as entrepreneurs looking to develop new applications based on Earth observation data; weather forecasters and presenters communicating about weather events and the climate; and weather- and climate-sensitive sectors of the economy, particularly through downstream processing of EUMETSAT data.
Starting at the bottom of the screen and just left of center, dust from the Sahara, seen as a hot pink color, moves toward the top right of the screen, over Europe, in February 2021. This animation is made with EUMETView, using the Meteosat Second Generation Dust Red Green Blue product
Medicane Ianos impacted Greece in mid-September 2020, killing four people and causing damage of more than Euro 80m euros. The animation, generated through EUMETView, shows the evolution of Medicane Ianos, a tropical-like cyclone that formed in the Mediterranean Sea. It comprises two layers, the precipitation at ground product, derived using infrared data from the geostationary orbit and microwave data from the low-Earth orbit, plus the MSG Natural Colour Red, Green Blue product from Meteosat-8, which flies at 41.5°E, over the Indian Ocean
How are the new services innovative?
EUMETSAT will provide an integrated and fully cloud-native service portfolio and a way of empowering the users of our data to choose and adapt the services to their needs. In this way, EUMETSAT is combining two concepts: the classic ‘data to the users’ with the modern ‘users to the data’.
The EUMETView service provides new functionalities for displaying visualizations and creating maps from across the EUMETSAT product range. EUMETView is an online map service that provides these visualizations through a customizable web user interface and an enhanced set of Open Geospatial Consortium (OGC) standard application programming interfaces (APIs).
Data viewing can be personalized by adding layers by satellite or theme, and the settings can be saved for future use. Personalization allows users to change the area of interest, map projection, base maps and overlays. Maps can be animated and downloaded, as can the corresponding georeferenced products.
Further functionality allows users to view data as point-based information and as time series. Maps may be saved in either ‘event mode’, tying the view to a specific point in time for later recall, or ‘live mode’, in which the latest updated version of the map will be recalled when requested. The product navigator stays up to date with the latest data sets that are available for viewing.
The APIs allow users to integrate EUMETView into other applications and systematically download images and products without accessing the web user interface. The EUMETSAT Data Store provides users with a single point of access to a growing catalog of EUMETSAT’s meteorological, climate and ocean data. Eventually, it will offer the full data catalog, including near-real-time and historical data and climate data records.
Access to the store is through a web user interface or a series of APIs. Products identified in the store can be combined with the functionality provided by the Data Tailor API, allowing for product customization. The Data Store allows users immediate access to the data, reducing delays. The product navigator will stay up to date with the latest data sets that are available for download.
The Data Tailor software makes it possible for users to subset and aggregate our data products in space and time, filter layers, generate quick looks, project onto new coordinate reference systems, and reformat into common GIS formats (netCDF, GeoTIFF, etc). It offers a uniform way to transform both historical and near-real-time satellite data provided by EUMETSAT.
What are the costs?
The Data Store and EUMETView services do not cost the users anything, unless they are commercial users and want to have access to our Meteosat data with a latency of less than three hours. For this kind of license, two types of flat rates apply: one for end users at €4,000 (US$4,711) per year and one for broadcasters/service providers at €8,000 (US$9,422) per year.
