Diving into the data deluge

CSIRO's Data61 launched a new report this week highlighting a $315B opportunity for the Australian economy over the next decade. The report outlines industry growth opportunities that ride on a third global wave of digital innovation enabled by a data-driven convergence of information and communication technologies.

I recently presented at an HPC industry event in August and in my welcome address referred to this data "deluge" and how it was impacting science. The following is an extract of my address:

__________

"Today, a tidal wave of data threatens to overwhelm us, consuming our attention by its sheer volume. In the space of a decade we have gone from a state of data deprivation to data deluge.

Sometimes it seems data is part of the problem, not the solution … the unwanted offspring of technologically advanced but unprepared and irresponsible parents.

This data deluge is changing our scientific approach – from a hypothesis-driven scientific method to science based on exploration.

Researchers no longer simply ask, “What experiment could I construct to test my hypothesis?”

They ask, “What conclusions can I draw from all this data?”

Or increasingly, “What insights could I uncover if I combine data from multiple disciplines and sources?”

It’s an exciting time for researchers. Supercomputing, artificial intelligence and machine learning are front and centre of this new wave of development.

The challenge is analysing petabytes of data on a time scale that is practical and useful in human terms.

The Square Kilometre Array, for example, will be one of the world's great scientific endeavours. Co-hosted by South Africa and Western Australia, the SKA telescopes will be 50 times more sensitive and 10,000 times faster than today’s most advanced telescopes. Construction of the SKA is scheduled to begin in 2019 for initial observations by 2022.

In the build-up to this incredible project, Australia has invested in two precursor telescopes – the Australian Square Kilometre Array Pathfinder and the Murchison Widefield Array.

From the remote Mid West region, data flows 800 kilometres to Perth, where it is processed by a powerful supercomputer called Galaxy at the Pawsey Supercomputing Centre.

Galaxy takes this data and converts it into science products, images and image cubes across a range of wave lengths and frequencies that scientists from across the globe can analyse and use to make their discoveries.

But data is impotent without good strategy.

Or to put it in commercial terms, data does not have of itself intrinsic value without the analytic filter that points toward discovery. 

As Gordon Bell reminds us in the introduction to Microsoft’s The Fourth Paradigm:

‘It was Tycho Brahe’s assistant Johannes Kepler who took Brahe’s catalog of systematic astronomical observations and discovered the laws of planetary motion. This established the division between the mining and analysis of captured and carefully archived experimental data and the creation of theories.’ 

The symbiotic relationship between data harvesting and value creation is fundamental.

The Pawsey Supercomputing Centre is at the centre of Australia’s endeavours to create meaningful value out of mountains of data, and deliver important discoveries across environmental, agricultural, medical and many other fields to benefit the world’s communities."