Services

CloudStor – a little history of a humble project that snowballed

Fields marked with an * are required

Subscribe to our newsletter

by AARNet’s eResearch Director, Guido Aben

Guido Aben will be presenting at the Terena conference (TNC2013) in Maastricht on 4 June 20

Transfer a Terabyte size file in around 5 hours: learn about the Terabyte Challenge

 

CloudStorThe history of CloudStor, or FileSender as the underlying software is called, starts somewhere in 2007. For a little context, in 2007 YouTube was 2 years old; Facebook had just turned three.

It wasn’t immensely apparent yet back then, but keen (some would say starry-eyed) observers could see the signs of the internet beginning to morph from being purely a consumption platform to a creation platform.

If you had real faith, and you squinted, you might even see the signs of browser builders adapting to this model, too. Still, Internet Explorer retained 80% market share in 2007.

Google Chrome didn’t exist yet (launched 2008). HTML5, the standard that today enables production and creativity using browsers, only got properly ratified in 2011, and even then only in part – it is still a living, moving standard today in 2013.

Browser as creation instrument?

In short, the idea of assuming that a browser, just a browser, would soon be a powerful creation instrument certainly was a little outlandish. Anybody who wanted to do something slightly funky using browsers used add-ins and applets; typically either Java or Adobe Flash.

In fact there was bit of a class distinction going on – commercial, slick applications preferred Flash, serious-minded, science types would naturally opt for Java applets. And so it certainly was in 2007; lots of research tools were written in Java, with interfaces looking like Java. It worked, but not as smoothly as YouTube.

Funding from DEEWR enabled progress

In 2007, AARNet received a pilot grant from the Department of Education Employment and Workplace Relations (DEEWR) to demonstrate the merits of connecting TAFE institutes to AARNet. Part of the money went to building real broadband enabled applications, one of which was a “large attachment inbox” for practitioners. It was well received, but definitely needed polishing.

Serendipitously, later that year during a meeting in Europe about R&E networks and storage, it turned out that a handful of other research networks had also tried their hand at such a “large attachment facility”; most of them because they could no longer bear the thought of users sending hard-disks and thumbdrives when a perfectly good network was available.

Global NREN collaboration

All of these projects were at a similar stage: they worked, but looked like escapees from a test lab. There and then, three of these networks – AARNet, as well as the Norwegian and Irish R&E networks, UNINETT and HEAnet– decided to meet this challenge head-on and do it properly. No copouts – it had to be delivered using nothing but a browser (no Java allowed!), and there couldn’t be a limit on the size of attachments, even though at the time no browser could deal with files over 2GB. Oh, and all of this on a shoestring budget, too.

It’s probably unnecessary to point out we’d lined up for a punishment banquet. Let’s just fast forward through a number of years battling absurd bugs, horribly fickle standards, undocumented features from major vendors, and take a look-in again around late 2009. That’s when the first beta service based on the new software went live.

AARNet took that plunge, ahead of the others, but our gleeful field reports made them follow suit, and by early 2010 we had all three project pioneers (AARNet, HEAnet and UNINETT) on a FileSender installation.

The rest of the R&E networking community soon caught on. By 2010, four others had joined; in 2011 six more joined, and in 2012 another 13 countries followed, including Internet2, our American counterpart, bringing us to a total of twenty-six sites.

Today, ten of those 26 sites contribute funding; nine of them have contributed code or man-hours; almost all have turned in translations in their local language. What started out as a shoestring project by a couple of NRENs (National Research and Education Networks) who had had enough is now a de-facto standard for file transfers in academic networks.

CloudStor usage is on the rise, making work easy for researchers

Domestic numbers are equally encouraging; where we started with perhaps a dozen new punters every month early in 2009, CloudStor now greets about 100 new users each week, and transports just shy of a Terabyte each week.

That’s what it’s all about: making life easier for our connected researchers. And we’re not done yet; 2013 is shaping up to be a good year for CloudStor, with features like multi-file upload and encryption in the pipeline.

The Terabyte Challenge discovers that it’s possible transfer a TB file in around 5 hours.

Another much-requested feature was faster uploads; the US research network Internet2  even challenged us to transfer a Terabyte in less than a business day. We took them up on the challenge, issued a Terabyte Challenge to a number of summer students, and sure enough, one team managed to put a number of tweaks and improvements into the base FileSender code that made it possible to almost saturate a 1 Gbit/s link and consequentially transfer a TB file in just under 5 hours.

This bodes well for a number of scientific disciplines that have seen an explosion in file sizes over the past few years – I’m thinking genomics in particular, where single genomic sequencer runs are already often over 100GB and showing no signs of slowed growth.

Up next and currently in beta release:

CloudStor+ (personal storage – 100GB free for eligible researchers).


Related Stories

Services / Videos

Mar 30, 2017

WSU teams up with AARNet and Zoom to build online collaborative culture

VIDEO INTERVIEW: Western Sydney University (WSU) partnered with AARNet and Zoom to deploy Zoom and Zoom Rooms across their eight campuses and has seen video collaboration increase by 400% while video conferencing costs dropped dramatically. We interviewed WSU CIO Kerry Holling and Senior Project Manager Geoff Lambert to learn more about the university's...

Education / Services / Teaching and Learning

Mar 22, 2017

CloudStor supports e-exams pilot project

NEW CASE STUDY: Despite the dominance of computers in our lives today, most exams across Australia are still completed using old-fashioned pen and paper. This can cause fatigue in candidates, see exam-markers expend unnecessary effort deciphering answers, and restrict curricula to topics that can be assessed by hand. Unhappy with...

Media Releases / Services

Mar 17, 2017

Panopto and AARNet Announce Video Platform Partnership

Agreement provides local Panopto video platform hosting and support for Australia's research and education community SYDNEY - Mar. 17, 2017 - Panopto and AARNet today announced a partnership to provide local hosting and support services for Panopto's leading video platform for education in Australia. Academic and research organisations based in Australia...