South Africa needs to consider the capacity needed to handle the massive amount of data that the Square Kilometre Array (SKA) project is expected to generate.
The precursor to the SKA, the MeerKAT, was recently launched in the Northern Cape, and streams of data are already pouring in, says Professor Russ Taylor, director of the Inter-University Institute for Data Intensive Astronomy (IDIA), addressing a Science Café in Grahamstown.
The combined dataset volumes from all the SKA pathfinders to date amount to approximately eight petabytes (8 000 TB) and researchers at IDIA expect this number to increase exponentially now that MeerKAT has joined the group.
“The MeerKAT telescope is the beginning of the creation of a data monster,” Taylor says.
The MeerKAT project involves several project scientists under the directorship of chief technologist Professor Justin Jonas, who is a Physics and Electronics Professor at Rhodes University.
The university has world-leading expertise in developing computing algorithms for solving the processing problems of MeerKAT and SKA data, and IDIA is currently in talks Rhodes about a partnership to facilitate the processing of this data.
Data sent to the Centre for High-Performance Computing (CHPC) in Cape Town via the telescope is currently shipped to the Centre for Radio Astronomy Techniques and Technologies (RATT) servers hosted by the Rhodes University’s IT Division, and IDIA’s research cloud for processing.
According to Taylor, data volumes are expected to grow by a factor of 10 million in Africa in the next 10 years.
“This is the biggest data challenge in science coming in the next decade,” he said. “And it’s coming here to South Africa.”
Taylor references “The Future: Six Drivers of Global Change” (Random House, 2013) in which Al Gore states, “… to develop a new generation of computer technology to store and process the data soon to be captured by the Square Kilometre Array, a new radio telescope that will collect each day twice the amount of information presently generated on the entire World Wide Web.”
Gore understood the challenges the SKA would present to the world in 2013, and these challenges are no less daunting today, especially for South Africa with the MeerKAT on local soil.
“Our challenge as South Africa is – if we’re going to do the science here, we need to be able to deal with the data. If we can’t deal with the data, we can’t do the science, and that means somebody else will,” Taylor warns.
The SKA aims to gather data to uncover fundamental mysteries about the universe, including the complete history of time, the nature of reality, and whether extraterrestrial life exists.