You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _editions/2025/tasks/multisumm.md
+31-22Lines changed: 31 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,49 +15,58 @@ blurb: "Participants are provided with multimodal web content from several citie
15
15
16
16
#### Task description
17
17
18
+
The goal of the task is to explore the creation of multimodal summaries from mulitple multimodal content items, specifically at MedaEval 2025, MultiSumm will explore the multimodal summarization of multiple websites. The
19
+
websites for summarization will be provided by the [H2020 Cultivate project](https://cultivate-project.eu/). CUltivate is exploring and creating resources for the promotion of Food Sharing Initiatives (FSIs) in urban and peri-urban environments for cities around world. A key element of the Cultuvate project is the creation of the ShareCity200 database. ShareCity200 will consist of an automatically crawled and curated database of the FSIs presents in 200 cities, primarily European, but also including cities from international territories beyond Europe. ShareCIty200 is an extension and exploration of the [ShrareCity100](https://sharecity.ie/research/sharecity100-database/) database created as part of an earlier project.
20
+
21
+
Participants in MultiSumm will be provided with the crawled FSI web content for a small number of selected cities, and asked to create a multimodal summary of the FSIs present in each city. Particpants will be provided with details of the requirements for the summaries and details of the summary evaluation methods to be used.
22
+
23
+
Since the ShareCity200 database will include details of FSIs in cities in many countries, and we are seeking to automate the evaluation process as much as possible, we will be open to including a specific cities at the request of individual participants to expand the linguistic scope of the task.
24
+
18
25
#### Motivation and background
19
26
27
+
Multidocument summarization for text documents has been a longstanding area of investigation. For example, for providing single summaries of multiple news articles on the same story. Traditionally this process has been complex and inflexible in terms of content style and test, requiring use of a wide variety of natural language processing (NLP) tools and detailed specification of the summarization process, The emergence of large language models (LLM) technologies has many revolutionised NLP tasks including summarization. The more recent arrival of multimodal LLMs is similarly impacting on topics relating to multimedia content.
28
+
29
+
While the MultiSumm tasks could be tackled using traditional NLP and multimedia processing tools, the expectation is that participants will tackle it using multimodal LLM methods. To the best of our knowledge, this will be the first benchmark task focusing on this topic and providing a potentialy valuable venue for exploration of the potential and challenges of use of multimodal LLMs in tasks of this sort.
30
+
20
31
#### Target group
21
32
22
-
#### Data
33
+
Researchers exploring use of multimodal LLMs, potentially drawn from both the NLP and multimedia research communities. One of the nice features of LLM methods is that they enable researchers to engage with tasks for which they are not expert with the methods and tools traditionally used to address them.
23
34
24
-
#### Ground truth
35
+
#### Data
25
36
26
-
#### Evaluation methodology
37
+
Data will be provided from that gathered in the Cultivate project. This will be the contents of crawled content from the open web.
38
+
Evaluation will make use of methods used for multidocument summarizatiomn, and evaluation methods current being develped for use in generative information retrieval (GenIR) application and methos being explored for the
39
+
use of LLM methods in automated evaluation of NLP tasks, the _LLM-as-Judge_ approach.
27
40
28
41
#### Quest for insight
29
42
Here are several research questions related to this challenge that participants can strive to answer in order to go beyond just looking at the evaluation metrics:
30
-
*<!-- # First research question-->
31
-
*<!-- # Second research question-->
32
-
<!-- # and so on-->
43
+
* What are the challenges of creating multi sources summaries of web content source?
44
+
* What are the most effective approaches to applying LLM methods in multimodal summarization?
45
+
* Identiying open research questions and challenges in applying LLM methods in multidocument summarization.
46
+
* What is the effectiveness of using LLM-based evaluation methods in multidocument summarization? answer in order to go beyond just looking at evaluation metrics.
33
47
34
48
#### Participant information
35
49
<!-- Please contact your task organizers with any questions on these points. -->
36
50
<!-- # * Signing up: Fill in the [registration form]() and fill out and return the [usage agreement](). -->
37
51
<!-- # * Making your submission: To be announced (check the task read me) <!-- Please add instructions on how to create and submit runs to your task replacing "To be announced." -->
38
52
<!-- # * Preparing your working notes paper: Instructions on preparing you working notes paper can be found in [MediaEval 2023 Working Notes Paper Instructions]().-->
53
+
More details will follow.
39
54
40
55
#### References and recommended reading
41
56
<!-- # Please use the ACM format for references https://www.acm.org/publications/authors/reference-formatting (but no DOI needed)-->
42
57
<!-- # The paper title should be a hyperlink leading to the paper online-->
58
+
More details will follow.
43
59
44
60
#### Task organizers
45
-
*<!-- # First organizer-->
46
-
*<!-- # Second organizer-->
47
-
<!-- # and so on-->
48
-
49
-
#### Task auxiliaries
50
-
<!-- # optional, delete if not used-->
51
-
*<!-- # First auxiliary-->
52
-
*<!-- # Second auxiliary-->
61
+
* Gareth J. F. Jones, DCU, Ireland
62
+
* Anastasia Potyagalova, DCU, Ireland
53
63
<!-- # and so on-->
54
64
55
65
#### Task schedule
56
-
* XX May 2025: Development Data release <!-- * XX May 2025: Data release <!-- # Replace XX with your date. We suggest setting the date in May - of course if you want to realease sooner it's OK. -->
57
-
* XX June 2025: Development Data release <!-- * XX June 2025: Data release <!-- # Replace XX with your date. We suggest setting the date in June - of course if you want to realease sooner it's OK. -->
58
-
* XX September 2025: Runs due and results returned. Exact dates to be announced. <!--* XX September 2025: Runs due <!-- # Replace XX with your date. We suggest setting enough time in order to have enough time to assess and return the results by the Results returned.-->
59
-
* 08 October 2025: Working notes paper <!-- Fixed. Please do not change.-->
60
-
* 25-26 October 2025: MediaEval Workshop, Dublin, Ireland and Online.. <!-- Fixed. Please do not change.-->
61
-
62
-
#### Acknowledgements
63
-
<!-- # optional, delete if not used-->
66
+
The program will be updated with the exact dates.
67
+
68
+
* May 2025: Development Data release
69
+
* June 2025: Development Data release
70
+
* September 2025: Runs due and results returned. Exact dates to be announced.
71
+
* 08 October 2025: Working notes paper
72
+
* 25-26 October 2025: MediaEval Workshop, Dublin, Ireland and Online.
0 commit comments