DAFx 2012 Evaluation

Our initial evaluation plan was to use surveys to gauge the reactions of participants to the training. However, as the DAFx tutorial was open to all conference attendees, we were unable to contact them directly for a survey. Evaluation was therefore largely based on the questions raised at DAFx and a debriefing session with the Sound Software team.

As mentioned in our initial post on DAFx, “the Dropbox question” was raised at DAFx for backups and questions on where to publish data. Digital audio research has no thematic repositories and few institutions currently have their own repositories. As such, there were few options for data publication for us to offer – for subsequent tutorials, we developed an extended list of possible ways to publish data including general online data sites (e.g. archive.org), research data publication sites (e.g. figshare.com), supplementary materials for journal articles and, if necessary, project web-sites.

Regarding the DropBox question, we people could use cloud storage for backups, but consideration should be given to the terms and conditions of service providers. Training materials have been updated to include specific references to the Google and Microsoft T&Cs, as these both give wide-ranging rights to the data holders. In addition, people were pointed to the JISC/DCC white paper on Curation In The Cloud for further information.

After the tutorial, feedback was exchanged with the Sound Software team, discussing which things worked as expected, and what we could do better. The lecture theatre used was quite deep, meaning that small text on slides was hard to read – especially for shortened web links, plausibly because they appear as fairly random series of characters (although the alternative option of very long URLs is not much better). An announcement at the start of the tutorial indicating that the slides, and therefore the links, would be made available online after the tutorial might have saved people trying to note down the links. Some consecutive slides were similar to each other meaning that it wasn't always obvious when slides changed – this was partly a technical issue as slides were created in Open Office, but were presented on PowerPoint with some formatting not appearing. Subsequent presentations were made from PDF versions of the slides for consistency.

Reactions at this conference varied from a complete disinterest in publishing research – which suggested that a stronger focus on future research council requirements, issues of data ownership and the risks affecting data might be required – to individual heads of research groups considering that they should be raising the profile of data management within their organisations. At an international conference, however, it is difficult to take into account various legislative strictures and the multitude of funder requirements. Selling data management to this audience requires personal motivations (carrots) rather than depending upon external forces (sticks). What was apparent, was that many researchers had not given consideration to what happened to their research in the longer term.

In order to engage more of the audience, we have researched online for cases of data loss and for papers relating to data loss. On the project wiki, we built a list of evidence to promote RDM [12] for use in materials. These included:

  • 2010/2011 Ponemon reports for Intel on lost laptops (ca. 10% of laptops in education and research being lost over a 3 year period – i.e. approx. the duration of a PhD)
  • a 2007 paper from E. Pinheiro, W.-D. Weber, and L.A. Barroso on Failure Trends in a Large Disk Drive Population was presented at the 5th USENIX Conference on File and Storage Technologies and showed ca. 13% of hard drives being replaced over a 3-year period – they are in fairly intensive use, but they have passed a burn-in period before being set live
  • some cloud storage terms and conditions give a large set of rights to the service providers e.g. Microsoft and Google
  • Credant Technologies found that “more than 17,000 USB sticks were left behind in 2010 in clothes left to be dry cleaned” [18]
  • news articles relating to research being lost in disasters (fire, flood, earthquake, hurricanes)
  • news articles relating to PhD theses-in-progress lost (largely through stolen laptops).

We then updated our materials to incorporate case studies of lost research (largely based on cases of lost PhDs reported on the internet) and statistics regarding hardware failures/losses.