Data-intensive research in domains such as radio astronomy, particle physics and the omics has the potential to generate an incredibly large volume of data. Some of the largest datasets on the planet are in the realm of 100’s of Petabytes!
Moving this data for analysis on high performance compute, or simply to use it for cross-institution collaborative research, requires high capacity frictionless networks that can perform over very long distances. Large single flows (dubbed “elephant flows”) have the potential to consume all available bandwidth, negatively impacting the performance of the network for other users.
IT managers are faced with the challenge of how to ensure “business as usual” functions of the many in academic and scientific institutions need to be as reliably served as the intensive demands of the few:
Much has already been done to date to enhance Australia’s National Research and Education Network, AARNet, and make this frictionless networking a reality.
AARNet engineers have been able to demonstrate that a little effort put into border architecture, data transfer tools and data handling workflows can result in extremely large data flows between research infrastructure services and instruments, without impacting on a broader range of users of the entire national network.
We are continuing to build new network infrastructure and new platforms and technologies to address this challenge and to support extreme, unique and evolving research institution requirements.
Brett’s role as AARNet Senior Research Analyst is to ensure that the research community learns to expect more from the significant long-term investment in the network and services provided by AARNet, and to understand that research networks worldwide are architected to serve even the most data-intensive sciences and collaborations.
Contact us to find out how we can help your institution manage big data flows.
May 25, 2020