https://multimediaeval.org/about/

1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
2<html xmlns="http://www.w3.org/1999/xhtml">
3	<head>
4		<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
5		<meta name="generator" content="RapidWeaver" />
6		
7		<title>About MediaEval</title>
8		<link rel="stylesheet" type="text/css" media="print" href="../rw_common/themes/simple/consolidated-print.css" />
9		<link rel="stylesheet" type="text/css" media="screen" href="../rw_common/themes/simple/consolidated-screen.css" />
10		<link rel="stylesheet" type="text/css" media="handheld" href="../rw_common/themes/simple/consolidated-handheld.css" />
11		
12		
13		
14		
15		
16		
17		
18		

18<script type="text/javascript" src="../rw_common/themes/simple/javascript.js"></script>

18
19		
20		
21		
22	</head>
23<body>
24<div id="container"><!-- Start container -->
25	
26	<div id="pageHeader"><!-- Start page header -->
27		
28		<h1>MediaEval Benchmarking Initiative for Multimedia Evaluation</h1>
29		<h2>The "multi" in multimedia: speech, audio, visual content, tags, users, context</h2>
30	</div><!-- End page header -->
31	
32	<div id="sidebarContainer"><!-- Start Sidebar wrapper -->
33		<div id="navcontainer"><!-- Start Navigation -->
34			<ul><li><a href="../" rel="self">Home</a></li><li><a href="./" rel="self" id="current">About MediaEval</a></li><li><a href="../datasets/" rel="self">Datasets</a></li><li><a href="../mediaeval2019/" rel="self">MediaEval 2019</a></li><li><a href="../mediaeval2018/" rel="self">MediaEval 2018</a></li><li><a href="../mediaeval2017/" rel="self">MediaEval 2017</a></li><li><a href="../mediaeval2016/" rel="self">MediaEval 2016</a></li><li><a href="../mediaeval2015/" rel="self">MediaEval 2015</a></li><li><a href="../mediaeval2014/" rel="self">MediaEval 2014</a></li><li><a href="../mediaeval2013/" rel="self">MediaEval 2013</a></li><li><a href="../mediaeval2012/" rel="self">MediaEval 2012</a></li><li><a href="../mediaeval2011/" rel="self">MediaEval 2011</a></li><li><a href="../mediaeval2010/" rel="self">MediaEval 2010</a></li><li><a href="../videoclef09/" rel="self">VideoCLEF 2009</a></li><li><a href="../videoclef08/" rel="self">VideoCLEF 2008</a></li><li><a href="../video/" rel="self">Videos about MediaEval</a></li><li><a href="../why/" rel="self">Why Participate?</a></li><li><a href="../who/" rel="self">Who are we?</a></li><li><a href="../openscience/" rel="self">Open Science</a></li><li><a href="../thanks/" rel="self">Acknowledgments</a></li></ul>
35		</div><!-- End navigation --> 
36		<div id="sidebar"><!-- Start sidebar content -->
37			<h1 class="sideHeader">MediaEval Bibliography</h1><!-- Sidebar header -->
38			We try to keep track of all the research results arising from MediaEval. A living bibliography of MediaEval work can be found at the <a href="http://www.citeulike.org/group/16499" rel="self">MediaEval group library on citeulike</a>. If you have written a publication using a MediaEval data set, please acknowledge the specific MediaEval task and also add it to the bibliography.<!-- sidebar content you enter in the page inspector -->
39			 <!-- sidebar content such as the blog archive links -->
40		</div><!-- End sidebar content -->
41	</div><!-- End sidebar wrapper -->
42	
43	<div id="contentContainer"><!-- Start main content wrapper -->
44		<div id="content"><!-- Start content -->
45			<strong>New MediaEval Website<br /></strong>This website was used from 2010-2019. For more up-to-date information see the <a href="https://multimediaeval.github.io/" rel="self">new MediaEval Website</a>.<strong><br /><br /></strong><strong>What is MediaEval?<br /></strong>MediaEval is a benchmarking initiative dedicated to evaluating new algorithms for multimedia access and retrieval. It emphasizes the 'multi' in multimedia and focuses on human and social aspects of multimedia tasks. MediaEval attracts participants who are interested in multimodal approaches to multimedia involving, e.g., speech recognition, multimedia content analysis, music and audio analysis, user-contributed information (tags, tweets), viewer affective response, social networks, temporal and geo-coordinates. <br /><br />For more information about past years see: <br /><br /><a href="https://youtu.be/3eE8A2mC2aw" rel="self">MediaEval 2017 Overview of the Year</a><br /><a href="http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7849098" rel="self">IEEE Multimedia Vol. 24, No. 1, 93-96, 2016 The Benchmarking Initiative for Multimedia Evaluation: MediaEval 2016</a><br /><a href="http://ercim-news.ercim.eu/en97/events/mediaeval-2013-evaluation-campaign" rel="self">ERCIM News 97, April 2014</a><br /><a href="http://www.signalprocessingsociety.org/technical-committees/list/sl-tc/spl-nl/2014-02/sltc-newsletter-february-2014-mediaeval-2013/" rel="self">IEEE Speech & Language Processing Technical Committee Newsletter Feb. 2014</a><br /><a href="http://records.sigmm.ndlab.net/2013/07/mediaeval-multimedia-benchmark-highlights-from-the-ongoing-2013-season/" rel="self">SIGMM Records Volume 5, Issue 2, June 2013</a><br /><a href="http://ercim-news.ercim.eu/en94/events/mediaeval-2012-evaluation-campaign" rel="self">ERCIM News 94, July 2013</a><br /><a href="http://ercim-news.ercim.eu/en88/events/mediaeval-2011-evaluation-campaign" rel="self">ERCIM News 88, January 2012</a><br /><a href="http://sigmm.org/records/records1102/featured01.html" rel="self">SIGMM Records </a><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; "><a href="http://sigmm.org/records/records1102/featured01.html" rel="self">Volume 3, Issue 2, June 2011</a></span><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; "><br /></span><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; "><a href="http://www.sigmm.org/records/records1002/featured03" rel="self">SIGMM Records Volume 2, Issue 2, June 2010</a></span><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; "><br /><br /></span><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; font-weight:bold; font-weight:bold; ">What is the MediaEval Workshop? </span><span style="font:12px &#39;Lucida Grande&#39;, LucidaGrande, Verdana, sans-serif; "><br /></span>The culmination of the yearly MediaEval benchmarking cycle is the MediaEval Workshop. The workshop brings together task participants to present their findings, discuss their approaches, learn from each other, and make plans for future work. The MediaEval Workshop co-located itself with ACM Multimedia conferences in 2010, 2013, and 2016 and with the European Conference on Computer Vision in 2012. It was an official satellite event of Interspeech conferences in 2011 and 2015. The workshop also welcomes attendees who did not participate in specific tasks, but who are interested in multimedia research, or getting involved in MediaEval in the future.<br /><br /><strong>How can I get involved?<br /></strong>MediaEval is an open initiative, meaning that any interested research group is free to signup and participate. Groups sign up for one or more tasks, they then receive task definitions, data sets and supporting resources, which they use to develop their algorithms. At the very end of the summer, groups submit their results and in the fall they attend the MediaEval workshop. See also <a href="../why/index.html" rel="self" title="Why Participate?">Why Participate?</a> or watch some video on the <a href="../video/index.html" rel="self" title="Videos about MediaEval">MediaEval video page</a>.<br /><br /><strong>What is a MediaEval task?</strong><br />A MediaEval task consists of four parts:<br /><ul class="square"><li>Data provided to the benchmark participants,</li><li>A task definition that describes the problem to be solved,</li><li>Ground truth against which participants&rsquo; algorithms are evaluated,</li><li>An evaluation metric. </li></ul>MediaEval tasks are oriented towards user needs in specific application settings and, to the extent possible, are based on scenarios of use derived from real-world problems.<br /><br /><strong>What is a MediaEval Task Force?<br /></strong>Proposing a task requires creating a task organization team, creating a task design (task definition that fits the user scenario, evaluation methodology) and laying the ground work for task logistics (source of data, source of ground truth, evaluation metric). MediaEval Task Forces are groups of people informally working towards a task to be proposed in a future year of MediaEval.<br /><br /><strong>Who runs MediaEval?</strong><br />MediaEval is a community-driven benchmark that is run by the MediaEval organizing committee consisting of the task organizers of all the individual task in a given year. The overall coordination is carried out by Martha Larson and Gareth Jones, who founded MediaEval in 2008 as VideoCLEF,  a track in the <a href="http://www.clef-campaign.org/" rel="self">CLEF Campaign</a>, and the other members of the MediaEval Community Council. Martha Larson serves as the overall contact person and the organizing force behind the MediaEval Workshop. MediaEval became an independent benchmarking initiative in 2010 under the auspices of the <a href="http://www.petamedia.eu/" rel="self">PetaMedia Network of Excellence</a>. In 2011, it also received support from <a href="http://eit.ictlabs.eu/" rel="self">ICT Labs</a> of <a href="http://eit.europa.eu/" rel="self">

45EIT</a>. It has also received support from various sources in order to offer student travel grants. We would particularly like to thank the <a href="http://elias-network.eu" rel="self">ELIAS</a> (Evaluating Information Access Systems), an ESF Research Networking Programme. Please refer to the pages of the individual years for complete lists of supporters. Since 2012, MediaEval has run as a fully bottom-up benchmark, in that it is not associated with a single "parent project".<br /><br /><strong>Can I get MediaEval data from past years? </strong>Many MediaEval tasks make <a href="../datasets/index.html" rel="self" title="Datasets">data available from past years</a>. Data release is made possible in cases where tasks are able to focus on creative commons content. Note that because the test data of one year often become the training data for the next, decisions about the release of the past year's data are often delayed until after the next year's tasks have been decided. <br /><br /><strong>How is MediaEval related to other benchmarks? <br /></strong>MediaEval is distinguished from other benchmarks by its focus on the social and human aspects of multimedia access and retrieval. This leads naturally to an emphasis on "the 'multi' of multimedia": the multitude of information sources that are available to tackle multimedia problem. The MediaEval community fosters collaboration and values bringing researchers together to work on new challenges. Although benchmarks, by their nature, involve quantitative comparison of systems. MediaEval also emphasizes qualitative insight into algorithms and techniques that participants develop. <br /><br />MediaEval takes great pains to complement rather than duplicate tasks running in other benchmarks, e.g., <a href="http://trecvid.nist.gov/" rel="self">TRECVid</a> video retrieval evaluation. MediaEval actively works to foster cooperation with other benchmarks, for example <a href="http://www.imageclef.org/" rel="self">ImageCLEF</a>, <a href="http://www.isical.ac.in/~fire/" rel="self">FIRE</a> and <a href="http://research.nii.ac.jp/ntcir/index-en.html" rel="self">NTCIR</a>. MediaEval is proud of its heritage as the VideoCLEF track of CLEF (Cross Language Evaluation Forum 2000-2009 and Conference and Labs of the Evaluation Forum 2010-present). For a historical picture of VideoCLEF in CLEF, check out the CLEF timeline graphic at <a href="http://www.clef-initiative.eu/track/series" rel="self">http://www.clef-initiative.eu/track/series</a><br /><br /><strong>How are the MediaEval tasks chosen? </strong><br />Each year there is an open call for new task proposals. Anyone who is interested in organizing a task can submit a call: usually task organizers are teams from multiple institutions. We chose tasks by first formulating a list of proposed tasks and then ascertaining which tasks are popular enough to run using a yearly survey. If you or your project would like to propose or organize a task in future years, please contact Martha Larson (m.a.larson at tudelft.nl).<br /><br /><strong>Are there intellectual property rights (IPR) issues with the data? </strong><br />MediaEval uses Creative Commons data wherever possible. When we need to make an exception we make effort to ensure that participants can license data through the appropriate channels.<br /><br />For more information contact Martha Larson m.a.larson (at) tudelft.nl.<br />
46		</div><!-- End content -->
47	</div><!-- End main content wrapper -->
48	
49	<div class="clearer"></div>
50	
51	<div id="footer"><!-- Start Footer -->
52		<div id="breadcrumbcontainer"><!-- Start the breadcrumb wrapper -->
53			
54		</div><!-- End breadcrumb -->
55		<p>&copy; 2020 MediaEval Multimedia Benchmark <a href="#" id="rw_email_contact">Contact</a>

55<script type="text/javascript">var _rwObsfuscatedHref0 = "mai";var _rwObsfuscatedHref1 = "lto";var _rwObsfuscatedHref2 = ":m.";var _rwObsfuscatedHref3 = "lar";var _rwObsfuscatedHref4 = "son";var _rwObsfuscatedHref5 = "@cs";var _rwObsfuscatedHref6 = ".ru";var _rwObsfuscatedHref7 = ".nl";var _rwObsfuscatedHref = _rwObsfuscatedHref0+_rwObsfuscatedHref1+_rwObsfuscatedHref2+_rwObsfuscatedHref3+_rwObsfuscatedHref4+_rwObsfuscatedHref5+_rwObsfuscatedHref6+_rwObsfuscatedHref7; document.getElementById('rw_email_contact').href = _rwObsfuscatedHref;</script>

55</p>
56	</div><!-- End Footer -->
57
58</div><!-- End container -->
59</body>
60</html>