CSIRO plans vast data hub for satellite data sharing

A data hub that’s served 20 petabytes of Earth observation data to Australasian users is set for an upgrade.

The Copernicus Australasia regional data hub provides free access to data collected by the satellites of the European Union’s Copernicus mission, covering Australasia, South-East Asia, the South Pacific, the Indian Ocean and the Australian Antarctic Territory.

There are currently three satellite series in the mission: Sentinel-1 (all-weather radar imaging), Sentinel-2 (high-resolution optical imaging) and Sentinel-3 (optical, radar and altimetry data), with missions planned through to Sentinel-6.

Australia’s data hub contains 5PB of data hosted on the National Computational Infrastructure, and since it went into operation in 2016, it’s served 20PB of downloads to users.

The CSIRO said that it's now been engaged by the Copernicus hub partners to “explore what is now possible for designing, building and operating” next-generation earth observation (EO) hubs.

From today’s 5PB, the partners expect 10PB of storage to be needed soon, with the ability to scale to over 30PB within 10 years.

New data management and access services are needed, and the CSIRO wants to understand what is “becoming possible” for storing EO data and providing “innovative EO data access services”.

The new hub could also host “authoritative replicas” of other data sets for local access.

The CSIRO is also canvassing private sector involvement in the next generation data hub, asking: “What are the many options for services and technologies from organisations for designing, building and operating the future hub located in Australia as an outsourced capability provided by Australian and global industry partners?"

However, even if outsourced the hub must be located “wholly within Australia”.

Partners in the hub consortium include Geoscience Australia, the Queensland, NSW and Western Australian state governments, Landgate, the European Commission, the European Space Agency, Eumetsat, the European Environment Agency, AARNET, and the National Computational Infrastructure.