Workshop report on Cloud-based Evaluation now available

The ‘Report on the Cloud-based Evaluation Approaches Workshop 2015’ by Henning Müller et al. is now available online:


Data analysis requires new approaches in many domains for evaluating tools and tech- niques, particularly when the data sets grow large and more complex. Evaluation–as–a– service (EaaS) was coined as a term to represent evaluation approaches based on APIs, virtual machines or source code submission, different from the common paradigm of evalu- ating techniques on a distributed test collection, tasks and submitted results files. Such new approaches become necessary when data sets become extremely large, contain confidential information or might change quickly over time. The workshop on cloud–based evaluation (CBE) took place in Boston, MA, USA on November 5, 2015 and explored several approaches for data analysis evaluation and frameworks in this field. The objective was to include several stakeholders from academic partners, companies to funding agencies to cover various inter- ests and viewpoints in the discussion of evaluation infrastructures. The workshop focused on the biomedical domain but the results are easily applicable to many domains of information analysis and retrieval.