Scientific papers

Alan F. Smeaton Alba Seco de Herrera; Bogdan Ionescu; Claire-Hélène Demarty; Faiyaz Doctor Graham Healy; Lorin Sweeney; Mihai Gabriel Constantin Rukiye Savran Kiziltepe;

Dublin City University; InterDigital Paris University of Essex; University Politehnica of Bucharest

Using a collection of publicly available links to short form video clips of an average of 6 seconds duration each, 1,275 users manually annotated each video multiple times to indicate both longterm and short-term memorability of the videos. The annotations were gathered as part of an online memory game and measured a participant’s ability to recall having seen the video previously when shown a collection of videos. The recognition tasks were performed on videos seen within the previous few minutes for short-term memorability and within the previous 24 to 72 hours for long-term memorability. Data includes the reaction times for each recognition of each video. Associated with each video are text descriptions (captions) as well as a collection of image-level features applied to 3 frames extracted from each video (start, middle and end). Video-level features are also provided. The dataset was used in the Video Memorability task as part of the MediaEval benchmark in 2020.

Open Access

Journal article

Data in Brief

Daniel Gatica-Perez; Mario Parra

Idiap Research Institute

In this study, we evaluated the feasibility of using zero-shot classification models for activity recognition in a Digital Sommelier. Our experiment involved preprocessing video data by extracting frames and categorization user activities related to a wine-tasting scenario. Image classification models demonstrated high accuracy, nearing 90%, in distinguishing between “engaged” and ” disengaged” stated. however, video classification models presented a lower performance in classifying user activities such as “observing wine”, “smelling wine” and “snipping wine”, with an average accuracy of around 50% due to the interdependent nature of activities. Despite these challenges, our findings highlight the potential of zero-shot classification models in enhancing virtual assistants’ ability to recognize and respond to user activities. 

Open Access

Publication

N/A

Mathias-Felipe de Lima-Santos Wilson Ceron

Universidade Federal de São Paulo; University of Amsterdam;

The information landscape has undergone significant transformations with the widespread adoption of the internet and online social networks. This has led to both positive and negative consequences. On the positive side, information can now spread quickly and reach a vast audience. Social media platforms have played a crucial role in fostering a culture of participation by motivating people to actively create and share content. However, there were also drawbacks. Social media platforms employ algorithms that restrict the diversity of content users are exposed to, leading to the reinforcement of pre-existing beliefs, commonly referred to as “echo chambers”

Open Access

Book section

Mapping Lies in the Global Media Spherre - Routlege

Alex Gomez-Villa Bartłomiej Twardowski Joost van de Weijer Marco Buzzelli Simone Zini

Autonomous University of Barcelona University of Florence; University of Milano- Bicocca

Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation – which we call Planckian Jitter – that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets.
In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations. Official code available at: https://github.com/TheZino/PlanckianJitter

Open Access

Conference paper

N/A

Hanna Lukashevich Jakob Abeßer Joachim Bös Sascha Grollmisch Sebastian Stober

Fraunhofer IDMT; Otto-von-Guericke University Magdeburg

Music classification algorithms use signal processing and machine learning approaches to extract and enrich metadata for audio recordings in music archives. Common tasks include music genre classification, where each song is assigned a single label (such as Rock, Pop, or Jazz), and musical instrument classification. Since music metadata can be ambiguous, classification algorithms cannot always achieve fully accurate predictions. Therefore, our focus extends beyond the correctly estimated class labels to include realistic confidence values for each potential genre or instrument label. In practice, many state-of-the-art classification algorithms based on deep neural networks exhibit overconfident predictions, complicating the interpretation of the final output values. In this work, we examine whether the issue of overconfident predictions and, consequently, non-representative confidence values is also relevant to music genre classification and musical instrument classification.
Moreover, we describe techniques to mitigate this behavior and assess the impact of deep ensembles and temperature scaling in generating more realistic confidence outputs, which can be directly employed in real-world music tagging applications.

Open Access

Conference paper Publication

Audio Mostly Conference

Alejandro Moreo; Berta Chulvi Paolo Rosso Silvia Corbara

ISTI-CNR; Scuola Normale Superiore Universitat Politècnica de Valènvia

Among the many tasks of the authorship field, Authorship Identification aims at uncovering the author of a document, while Author Profiling focuses on the analysis of personal characteristics of the author(s), such as gender, age, etc. Methods devised for such tasks typically focus on the style of the writing, and are expected not to make inferences grounded on the topics that certain authors tend to write about. In this paper, we present a series of experiments evaluating the use of topic- agnostic feature sets for Authorship Identification and Author Profiling tasks in Spanish political language. In particular, we propose to employ features based on rhythmic and psycholinguistic patterns, obtained via different approaches of text masking that we use to actively mask the underlying topic. We feed these feature sets to a SVM learner, and show that they lead to results that are comparable to those obtained by a BETO transformer, when the latter is trained on the original text, i.e., potentially learning from topical information. Moreover, we further investigate the results for the different authors, showing that variations in performance are partially explainable.

Open Access

Conference paper

N/A

Alberto Del Bimbo; Andrea Ciamarra Federico Becattini; Lorenzo Seidenari;

University of Florence;

Forecasting motion and spatial positions of objects is of fundamental importance, especially in safety-critical settings such as autonomous driving. In this work, we address the issue by forecasting two different modalities that carry complementary information, namely optical flow and depth. To this end we propose FLODCAST a flow and depth forecasting model that leverages a multitask recurrent architecture, trained to jointly forecast both modalities at once. We stress the importance of training using flows and depth maps together, demonstrating that both tasks improve when the model is informed of the other modality. We train the proposed model to also perform predictions for several timesteps in the future. This provides better supervision and leads to more precise predictions, retaining the capability of the model to yield outputs autoregressively for any future time horizon. We test our model on the challenging Cityscapes dataset, obtaining state of the art results for both flow and depth forecasting. Thanks to the high quality of the generated flows, we also report benefits on the downstream task of segmentation forecasting, injecting our predictions in a flow-based mask-warping framework.

Open Access

Journal article

Pattern Recognition Letters

Christos Koutlis; Giorgios Kordopatis-Zilos; Ioannis Kompatsiaris; Ioannis Sarridis Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

In this paper, we introduce InDistill, a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student.  This is achieved through a curriculum learning-based training scheme that considers the distillation difficulty of each layer and the critical learning periods when the information flow paths are established. This procedure can lead to a student model that is better prepared to learn from the teacher. To ensure the applicability of InDistill across a wide range of teacher-student pairs, we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher’s intermediate layers to match those of the student, allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10, CIFAR-100, and ImageNet datasets showcasing that preserving the information flow paths consistently increases the performance of the baseline KD approaches on both classification and retrieval settings.

Open Access

Conference paper

N/A

Christos Papaioannidis Ioanna Valsamara; Ioannis Pitas;

Aristotle University of Thessaloniki;

Recently, multi-agent systems that facilitate knowledge sharing among Deep Neural Network (DNN) agents, have gained increasing attention. This paper explores the dynamics of multi-agent systems that support Teacher-Student DNN interactions, where knowledge is distilled from Teachers to Students. Within such systems, selecting the most compatible Teacher for a given task is far from trivial and can lead to low-quality decisions. Hence, the need arises for accurate domain knowledge evaluation. In that context, we propose including an OOD detection module in each DNN agent to enable effective agent expertise evaluation and precise identification of suitable Teachers. This setup allows Student agents to distill knowledge from the most knowledgeable Teachers within a specific domain, ensuring optimal system performance. To effectively utilize OOD detection in this context, we address key challenges such as determining the minimum data cardinality required to ensure optimal performance and reliable inferences of the OOD detectors.

Open Access

Paper Publication Research article

N/A

Christos Papaioannidis Ioanna Valsamara; Ioannis Pitas;

Aristotle University of Thessaloniki;

In today’s data-driven world, the exponential growth of data across various sectors presents unique opportunities and challenges. In this paper, we propose a novel method tailored to enhance the efficiency of Deep Neural Networks (DNNs) in managing these vast data amounts. The primary challenge addressed is the ability of DNNs to provide inferences on the minimal amount of data without sacrificing their quality, a significant concern given the vast scales involved in big data analytics. Our approach emphasizes DNN inference efficiency and reliability, enabling DNNs to deliver accurate inferences while substantially reducing computational complexity. This study explores the increasingly attractive deployment of DNNs for complex tasks, focusing on determining the minimal amount of data necessary to ensure optimal network performance and reliable inference outputs, improving the applicability of DNNs across various big data environments.

Open Access

Paper Publication Research article

N/A

Dimitrios Papaioannou; Ioannis Pitas; Vasileios Mygdalis

Aristotle University of Thessaloniki; University of Antwerp

In the realm of machine learning systems, achieving consensus among networking nodes is a fundamental yet challenging task. This paper presents Proof of Quality Inference (PoQI), a novel consensus protocol designed to integrate deep learning inference under the basic format of the Practical Byzantine Fault Tolerant (P-BFT) algorithm. PoQI is applied to Deep Neural Networks (DNNs) to infer the quality and authenticity of produced estimations by evaluating the trustworthiness of the DNN node’s decisions. In this manner, PoQI enables DNN inference nodes to reach a consensus on a common DNN inference history in a fully decentralized fashion, rather than relying on a centralized inference decision-making process. Through PBFT adoption, our method ensures byzantine fault tolerance, permitting DNN nodes to reach an agreement on inference validity swiftly and efficiently. We demonstrate the efficacy of PoQI through theoretical analysis and empirical evaluations, highlighting its potential to forge trust among unreliable DNN nodes.

Open Access

Paper Preprint Publication

N/A

Antidio Viguria Francisco Pérez-Grau Ioannis Pitas; Marco Montes-Grova Vasileios Mygdalis

Aristotle University of Thessaloniki;

Novel view synthesis is the task of generating new images that render an object or scene from a different viewpoint than the one given. It aims to create new views of a specific subject starting from a number of pictures taken from known points of view. The novel view synthesis problem can be approached in two different ways: as a problem of interpolation of images between two known images or extrapolation of images from one or a subset of images. During this work, the problem of the extrapolation will be addressed, taking advantage of the fact that it is possible to pre-calculate the trajectories that we want the camera that takes the images to execute, from a series of known shot-types. Based on that and on the Autoregressive Transformers, it is presented an end-to-end tool for novel-view synthesis from previously unvisited points of view for aerial cinematography robots.

Open Access

Paper Preprint Publication

N/A

Anestis Christidis Christos Papaioannidis Ioannis Mademlis; Ioannis Pitas;

Aristotle University of Thessaloniki;

Human gesture recognition is a very important tool in human-computer or human-robot interaction. In many cases, such algorithms may need to be executed on systems with limited computational capabilities, due to size or weight constraints, introducing restrictions that can impede gesture recognition performance. This paper proposes a gesture recognition method that is based on a very simple and lightweight Deep Neural Network (DNN) architecture, suitable for embedded execution. In order to achieve increased accuracy without a large computational/memory overhead, the proposed method utilizes as input both full 2D human body skeletons and image patches extracted from regions of interest (e.g., around human arms) in each video frame. These two input types are processed in parallel by separate modules and the corresponding features are fused before being exploited for gesture recognition. Reliance on 2D skeleton sequences allows the utilization of a lightweight DNN architecture, while the image patches convey rich semantic information that enhances gesture recognition performance. This approach is unlike existing similar methods, which only exploit skeleton sequences. Experimental evaluation indeed shows increased recognition accuracy, indicating that the proposed method offers a reliable solution for human gesture recognition on embedded systems.

Open Access

Paper Preprint Publication

N/A

Claudio Gennaro; Fabrizio Falchi; Gabriele Lagani; Giuseppe Amato;

ISTI-CNR;

Recent work on sample efficient training of Deep Neural Networks (DNNs) proposed a semi-supervised methodology based on biologically inspired Hebbian learning, combined with traditional backprop-based training. Promising results were achieved on various computer vision benchmarks, in scenarios of scarce labeled data availability. However, current Hebbian learning solutions can hardly address large-scale scenarios due to their demanding computational cost. In order to tackle this limitation, in this contribution, we investigate a novel solution, named FastHebb (FH), based on the reformulation of Hebbian learning rules in terms of matrix multiplications, which can be executed more efficiently on GPU. Starting from Soft-Winner-Takes-All (SWTA) and Hebbian Principal Component Analysis (HPCA) learning rules, we formulate their improved FH versions: SWTA-FH and HPCA-FH. We experimentally show that the proposed approach accelerates training speed up to 70 times, allowing us to gracefully scale Hebbian learning experiments on large datasets and network architectures such as ImageNet and VGG.

Open Access

Journal article

Carlos Santiago; Claudio Gennaro; Fabrizio Falchi; Giuseppe Amato; Luca Ciampi;

Institute of Information Science and Technologies Instituto Superior Técnico;

This work addresses the challenge of video violence detection in data-scarce scenarios, focusing on bridging the domain gap that often hinders the performance of deep learning models when applied to unseen domains. We present a novel unsupervised domain adaptation (UDA) scheme designed to effectively mitigate this gap by combining supervised learning in the train (source) domain with unlabeled test (target) data. We employ single-image classification and multiple instance learning (MIL) to select frames with the highest classification scores, and, upon this, we exploit UDA techniques to adapt the model to unlabeled target domains. We perform an extensive experimental evaluation, using general-context data as the source domain and target domain datasets collected in specific environments, such as violent/non-violent actions in hockey matches and public transport. The results demonstrate that our UDA pipeline substantially enhances model performances, improving their generalization capabilities in novel scenarios without requiring additional labeled data.

Open Access

Journal article

SN Computer Science

Alberto Del Bimbo; Federico Becattini; Francesco Marchetti; Lorenzo Seidenari; Lucile Sassatelli; Quentin Guimard

Institut Universitaire de France; Université Côte d'Azur; University of Florence;

Prediction of head movements in immersive media is key to designing efficient streaming systems able to focus the bandwidth budget on visible areas of the content. However, most of the numerous proposals made to predict user head motion in 360° images and videos do not explicitly consider a prominent characteristic of the head motion data: its intrinsic uncertainty. In this article, we present an approach to generate multiple plausible futures of head motion in 360° videos, given a common past trajectory. To our knowledge, this is the first work that considers the problem of multiple head motion prediction for 360° video streaming. We introduce our discrete variational multiple sequence (DVMS) learning framework, which builds on deep latent variable models. We design a training procedure to obtain a flexible, lightweight stochastic prediction model compatible with sequence-to-sequence neural architectures. Experimental results on 4 different datasets show that our method DVMS outperforms competitors adapted from the self-driving domain by up to 41% on prediction horizons up to 5 sec., at lower computational and memory costs. To understand how the learned features account for the motion uncertainty, we analyze the structure of the learned latent space and connect it with the physical properties of the trajectories. We also introduce a method to estimate the likelihood of each generated trajectory, enabling the integration of DVMS in a streaming system. We hence deploy an extensive evaluation of the interest of our DVMS proposal for a streaming system. To do so, we first introduce a new Python-based 360° streaming simulator that we make available to the community. On real-world user, video, and networking data, we show that predicting multiple trajectories yields higher fairness between the traces, the gains for 20 to 30% of the users reaching up to 10% in visual quality for the best number K of trajectories to generate.

Open Access

Journal article

ACM Transactions on Multimedia Computing, Communications, and Applications

Claudio Gennaro; Giuseppe Amato; Lucia Vadicamo;

ISTI-CNR;

Permutation-based Indexing (PBI) approaches have been proven to be particularly effective for conducting large-scale approximate metric searching. These methods rely on the idea of transforming the original metric objects into permutation representations, which can be efficiently indexed using data structures such as inverted files.
The standard conceptualization of permutation associated with a metric object involves only the use of object distances and their relative orders from a set of anchors called pivots. In this paper, we generalized this definition in order to enlarge the class of permutation representations that can be used by PBI approaches. In particular, we introduced the concept of permutation induced by a space transformation and a sorting function, and we investigated which properties these transformations should possess to produce permutations that are effective for metric search. Furthermore, as a practical outcome, we defined a new type of permutation representation that is calculated using distances from pairs of pivots. This proposed technique allowed us to produce longer permutations than traditional ones for the same number of object pivot distance calculations. The advantage lies in the fact that when longer permutations are employed, the use of inverted files built on permutation prefixes leads to greater efficiency in the search phase.

Open Access

Journal article

N/A

Georgios Tzimiropoulos; Ioannis Maniadis Metaxas; Ioannis Patras;

Queen Mary University of London;

Self-supervised learning has recently emerged as the preeminent pretraining paradigm across and between modalities, with remarkable results. In the image domain specifically, group (or cluster) discrimination has been one of the most successful methods. However, such frameworks need to guard against heavily imbalanced cluster assignments to prevent collapse to trivial solutions. Existing works typically solve this by reweighing cluster assignments to promote balance, or with offline operations (e.g. regular re-clustering) that prevent collapse. However, the former typically requires large batch sizes, which leads to increased resource requirements, and the latter introduces scalability issues with regard to large datasets. In this work, we propose ExCB, a framework that tackles this problem with a novel cluster balancing method. ExCB estimates the relative size of the clusters across batches and balances them by adjusting cluster assignments, proportionately to their relative size and in an online manner. Thereby, it overcomes previous methods’ dependence on large batch sizes and is fully online, and therefore scalable to any dataset. We conduct extensive experiments to evaluate our approach and demonstrate that ExCB: a) achieves state-of-the-art results with significantly reduced resource requirements compared to previous works, b) is fully online, and therefore scalable to large datasets, and c) is stable and effective even with very small batch sizes.

Open Access

Conference paper

European Conference on Computer Vision

Chen Feng; Georgios Tzimiropoulos; Ioannis Patras;

Queen Mary University of London;

Despite the large progress in supervised learning with neural networks, there are significant challenges in obtaining high-quality, large-scale and accurately labelled datasets. In such contexts, how to learn in the presence of noisy labels has received more and more attention. Addressing this relatively intricate problem to attain competitive results predominantly involves designing mechanisms that select samples that are expected to have reliable annotations. However, these methods typically involve multiple off-the-shelf techniques, resulting in intricate structures. Furthermore, they frequently make implicit or explicit assumptions about the noise modes/ratios within the dataset. Such assumptions can compromise model robustness and limit its performance under varying noise conditions. Unlike these methods, in this work, we propose an efficient and effective framework with minimal hyperparameters that achieves SOTA results in various benchmarks. Specifically, we design an efficient and concise training framework consisting of a subset expansion module responsible for exploring non-selected samples and a model training module to further reduce the impact of noise, called NoiseBox . Moreover, diverging from common sample selection methods based on the “small loss” mechanism, we introduce a novel sample selection method based on the neighbouring relationships and label consistency in the feature space. Without bells and whistles, such as model co-training, self-supervised pre-training and semi-supervised learning, and with robustness concerning the settings of its few hyper-parameters, our method significantly surpasses previous methods on both CIFAR10/CIFAR100 with synthetic noise and real-world noisy datasets such as Red Mini-ImageNet, WebVision, Clothing1M and ANIMAL-10N.

Open Access

Journal article

N/A

Andrea Esuli; Claudio Gennaro; Davide Alessandro Coccomini; Fabrizio Falchi; Giuseppe Amato;

ISTI-CNR;

Open Access

Publication

N/A

Jean De Meyere Noémie Krack

KU Leuven

Open Access

Conference paper

International Congress towards a responsible development of Metaverse

Christos Koutlis; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

Open Access

Conference paper

European Conference on Computer Vision

Ioannis Kompatsiaris; John Violos Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

This paper discusses four facets of the Knowledge Distillation (KD) process for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures, particularly when executed on edge devices with constrained processing capabilities. First, we conduct a comparative analysis of the KD process between CNNs and ViT architectures, aiming to elucidate the feasibility and efficacy of employing different architectural configurations for the teacher and student, while assessing their performance and efficiency. Second, we explore the impact of varying the size of the student model on accuracy and inference speed, while maintaining a constant KD duration. Third, we examine the effects of employing higher resolution images on the accuracy, memory footprint and computational workload. Last, we examine the performance improvements obtained by fine-tuning the student model after KD to specific downstream tasks. Through empirical evaluations and analyses, this research provides AI practitioners with insights into optimal strategies for maximizing the effectiveness of the KD process on edge devices.

Open Access

Conference paper

Signal Processing and Communication

Hannes Fassold;

Joanneum Research;

Open Access

Publication

Conference on Imaging, Signal Processing and Communication

Claudio Gennaro; Claudio Vairo; Fabio Carrara; Fabrizio Falchi; Giuseppe Amato; Lucia Vadicamo; Nicola Messina; Paolo Bolettieri;

ISTI-CNR;

VISIONE is a versatile video retrieval system supporting diverse search functionalities, including free-text, similarity, and temporal searches. Its recent success in securing first place in the 2024 Video Browser Showdown (VBS) highlights its effectiveness.  
Originally designed for analyzing, indexing, and searching diverse video content, VISIONE can also be adapted to images from lifelog cameras thanks to its reliance on frame-based representations and retrieval mechanisms.

In this paper, we present an overview of VISIONE’s core characteristics and the adjustments made to accommodate lifelog images. These adjustments primarily focus on enhancing result visualization within the GUI, such as grouping images by date or hour to align with lifelog dataset imagery. It’s important to note that while the GUI has been updated, the core search engine and visual content analysis components remain unchanged from the version presented at VBS 2024. Specifically, metadata such as local time, GPS coordinates, and concepts associated with images are not indexed or utilized in the system. Instead, the system relies solely on the visual content of the images, with date and time information extracted from their filenames, which are utilized exclusively within the GUI for visualization purposes.

Our objective is to evaluate the system’s performance within the Lifelog Search Challenge, emphasizing reliance on visual content analysis without additional metadata.

Open Access

Conference paper

Bergman Clement Frédéric Precioso Julie Tores Léa Andolfi Lucile Sassatelli; Magali Guaresi Sarah Lecossais Thierry Devars Victor Ecrement Virginie Julliard Wu Hui-Yin

Inria; Institut Universitaire de France; Sorbonne Université Université Côte d'Azur; Université Sorbonne Paris Nord

In film gender studies the concept of “male gaze” refers to the way the characters are portrayed on-screen as objects of desire rather than subjects. In this article we introduce a novel video-interpretation task to detect character objectification in films. The purpose is to reveal and quantify the usage of complex temporal patterns operated in cinema to produce the cognitive perception of objectification. We introduce the ObyGaze12 dataset made of 1914 movie clips densely annotated by experts for objectification concepts identified in film studies and psychology. We evaluate recent vision models show the feasibility of the task and where the challenges remain with concept bottleneck models. Our new dataset and code are made available to the community.

Open Access

Conference paper Publication

IEEE Conference on Computer Vision and Pattern Recognition

Tobias Blanke;

University of Amsterdam;

Open Access

Journal article

N/A

Adrian Popescu; Bogdan Ionescu; Cristian Stanciu; Giorgios Kordopatis-Zilos; Luca Cuccovillo; Roberto Cardelli; Symeon Papadopoulos

CEA; CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; Fraunhofer IDMT; Mercatorum University; University Politehnica of Bucharest

Front matter of the proceedings of the 3nd ACM International Workshop on Multimedia AI against Disinformation, held in Phuket (Thailand) on June 10th, 2024.

The full proceedings are available online at https://dl.acm.org/doi/proceedings/10.1145/3643491.

Open Access

Book section

ACM Association for Computing Machinery

Adrian Popescu; Bogdan Ionescu; Cristian Stanciu; Giorgios Kordopatis-Zilos; Luca Cuccovillo; Roberto Cardelli; Symeon Papadopoulos

CEA; CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; Fraunhofer IDMT; Mercatorum University; University Politehnica of Bucharest

Synthetic media generation and manipulation have seen rapid advancements in recent years, making it increasingly easy to create multimedia content that is indistinguishable to the human observer. Moreover, generated content can be used maliciously by individuals and organizations in order to spread disinformation, posing a significant threat to society and democracy. Hence, there is an urgent need for AI tools geared towards facilitating a timely and effective media verification process. The MAD’24 workshop seeks to bring together people with diverse backgrounds who are dedicated to combating disinformation in multimedia through the means of AI, by fostering an environment for exploring innovative ideas and sharing experiences. The research areas of interest encompass the identification of manipulated or generated content, along with the investigation of the dissemination of disinformation and its societal repercussions. Recognizing the significance of multimedia, the workshop emphasizes the joint analysis of various modalities within content, as verification can be improved by aggregating multiple forms of content. 

Open Access

Conference paper

ACM on Multimedia Retrieval

Evlampios Apostolidis; Konstantinos Tsigos Spyridon Baxevanakis; Symeon Papadopoulos Vasileios Mezaris

CERTH - Center for Research and Technology Hellas

In this paper we propose a new framework for evaluating the performance of explanation methods on the decisions of a deepfake detector. This framework assesses the ability of an explanation method to spot the regions of a fake image with the biggest influence on the decision of the deepfake detector, by examining the extent to which these regions can be modified through a set of adversarial attacks, in order to flip the detector’s prediction or reduce its initial prediction; we anticipate a larger drop in deepfake detection accuracy and prediction, for methods that spot these regions more accurately. Based on this framework, we conduct a comparative study using a state-of-the-art model for deepfake detection that has been trained on the FaceForensics++ dataset, and five explanation methods from the literature. The findings of our quantitative and qualitative evaluations document the advanced performance of the LIME explanation method against the other compared ones, and indicate this method as the most appropriate for explaining the decisions of the utilized deepfake detector.

Open Access

Conference paper

ACM on Multimedia Retrieval

Evlampios Apostolidis; Ioannis Kontostathis; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas

In this paper we introduce a new dataset for 360-degree video summarization: the transformation of 360-degree video content to concise 2D-video summaries that can be consumed via traditional devices, such as TV sets and smartphones. The dataset includes ground-truth human-generated summaries, that can be used for training and objectively evaluating 360-degree video summarization methods. Using this dataset, we train and assess two state-of-the-art summarization methods that were originally proposed for 2D-video summarization, to serve as a baseline for future comparisons with summarization methods that are specifically tailored to 360-degree video. Finally, we present an interactive tool that was developed to facilitate the data annotation process and can assist other annotation activities that rely on video fragment selection.

Open Access

Conference paper

Davide Alessandro Coccomini; Fabrizio Falchi; Giorgios Kordopatis-Zilos; Giuseppe Amato; Roberto Caldelli Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; ISTI-CNR;

In this paper, we present MINTIME, a video deepfake detection method that effectively captures spatial and temporal inconsistencies in videos that depict multiple individuals
and varying face sizes. Unlike previous approaches that either employ simplistic a-posteriori aggregation schemes, i.e., averaging or max operations, or only focus on the largest face in the video, our proposed method learns to accurately detect spatio-temporal inconsistencies across multiple identities in a video through a Spatio-Temporal Transformer combined with a Convolutional Neural Network backbone. This is achieved through an Identityaware Attention mechanism that applies a masking operation on the face sequence to process each identity independently, which enables effective video-level aggregation. Furthermore, our system incorporates two novel embedding schemes: (i) the Temporal Coherent Positional Embedding, which encodes the temporal information of the face sequences of each identity, and (ii) the Size Embedding, which captures the relative sizes of the faces to the video frames. MINTIME achieves state-of-the-art performance on the ForgeryNet dataset, with a remarkable improvement of up to 14% AUC in videos containing multiple people. Moreover, it demonstrates very robust generalization capabilities in crossforgery and cross-dataset settings. The code is publicly available at: https://github.com/davide-coccomini/MINTIME-Multi-Identity-size-iNvariant-TIMEsformer-for-Video-Deepfake-Detection

Open Access

Journal article

N/A

Cathal Gurrin; Fabio Carrara; Florian Spiess; Jakub Lokoč; Klaus Schoeffmann; Ladislav Peška; Loris Sauter; Luca Rossetto; Lucia Vadicamo; Minh-Triet Tran Nico Hezel Nicola Messina; Rahel Arnold; Sebastian Lubos Stefanos Vrochidis; Thao-Nhu Nguyen; Werner Bailer; Xingham Li Zhixin Ma;

CERTH - Center for Research and Technology Hellas Charles University; Dublin City University; HTW Berlin; Institute of Information Science and Technologies ISTI-CNR; Joanneum Research; University of Basel; University of Klagenfurt University of Zurich; Vietnam National University Wuhan University

This paper conducts a thorough examination of the 12th Video Browser Showdown (VBS) competition, a well-established international benchmarking campaign for interactive video search systems. 
The annual VBS competition has witnessed a steep rise in the popularity of multimodal embedding-based approaches in interactive video retrieval. Most of the thirteen systems participating in VBS 2023 utilized a CLIP-based cross-modal search model, allowing the specification of free-form text queries to search visual content.  This shared emphasis on joint embedding models contributed to balanced performance across various teams. However, the distinguishing factors of the top-performing teams included the adept combination of multiple models and search modes, along with the capabilities of interactive interfaces to facilitate and refine the search process.
Our work provides an overview of the state-of-the-art approaches employed by the participating systems and conducts a thorough analysis of their search logs, which record user interactions and results of their queries for each task. Our comprehensive examination of the VBS competition offers assessments of the effectiveness of the retrieval models, browsing efficiency, and user query patterns. Additionally, it provides valuable insights into the evolving landscape of interactive video retrieval and its future challenges.

Open Access

Journal article

IEEE Access

Emmanouil Krasanakis; Ioannis Kompatsiaris; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

Open Access

Publication

N/A

Albin Soutif-Cormerais Andrew Bagdanov Joost van de Weijer Simone Magistri Tommaso Trinci

Computer Vision Center University of Florence;

Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from a sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, which results in feature drift which is difficult to compensate for in the exemplar-free setting. To address this problem, we propose a simple and effective approach that consolidates feature representations by regularizing drift in directions highly relevant to previous tasks and employs prototypes to reduce task-recency bias. Our method, called Elastic Feature Consolidation (EFC), exploits a tractable second-order approximation of feature drift based on an Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes used in a novel asymmetric cross entropy loss which effectively balances prototype rehearsal with data from new tasks. Experimental results on CIFAR-100, Tiny-ImageNet, ImageNet-Subset and ImageNet-1K demonstrate that Elastic Feature Consolidation is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-the-art.

Open Access

Conference paper

N/A

Antonios Liapis; Georgios N. Yannakakis; Marvin Zammit;

University of Malta

Open Access

Conference paper

N/A

Hannes Fassold;

Joanneum Research;

Open Access

Conference paper

N/A

Connor Richard Lucia Vadicamo;

ISTI-CNR; University of St. Andrews

Dimensionality reduction techniques map values from a high dimensional space to one with a lower dimension. The result is a space which requires less physical memory and has a faster distance calculation. These techniques are widely used where required properties of the reduced-dimension space give an acceptable accuracy with respect to the original space. Many such transforms have been described. They have been classified in two main groups: linear and topological. Linear methods such as Principal Component Analysis (PCA) and Random Projection (RP) define matrix-based transforms into a lower dimension of Euclidean space. Topological methods such as Multidimensional Scaling (MDS) attempt to preserve higher-level aspects such as the nearest-neighbour relation, and some may be applied to non-Euclidean spaces. Here, we introduce nSimplex Zen, a novel topological method of reducing dimensionality. Like MDS, it relies only upon pairwise distances measured in the original space. The use of distances, rather than coordinates, allows the technique to be applied to both Euclidean and other Hilbert spaces, including those governed by Cosine, Jensen–Shannon and Quadratic Form distances. We show that in almost all cases, due to geometric properties of high-dimensional spaces, our new technique gives better properties than others, especially with reduction to very low dimensions.

Open Access

Journal article

ACM Transactions on Knowledge Discovery from Data

Bogdan Ionescu; Mihai Gabriel Constantin

University Politehnica of Bucharest

Video memorability is one of the vital aspects of subjective multimedia perception and, as such, is closely and thoroughly studied in the computer vision literature. This paper presents the methods proposed by AIMultimediaLab for the generalization subtask of the 2023 edition of the Predicting Video Memorability task. We explore several methods for augmenting the training process for a video Vision Transformer network, aiming to increase the number of hard-to-predict samples in the training set in order to increase the robustness of the targeted AI model. Starting from our previous works, we analyze several visual features that define “hard-to-predict” samples, and based on these features, we augment the training data of our models to target those specific videos that pose problems for memorability prediction.

Open Access

Conference paper

MediaEval

Claudio Vairo; Fabio Carrara; Jakub Lokoč; Kai Uwe Barthel; Klaus Schoeffmann; Konstantin Schall; Ladislav Peška; Lucia Vadicamo; Werner Bailer;

HTW Berlin; ISTI-CNR; Joanneum Research; University of Klagenfurt

CLIP-based text-to-image retrieval has proven to be very effective at the interactive video retrieval competition Video Browser Showdown 2022, where all three top-scoring teams had implemented a variant of a CLIP model in their system. Since the performance of these three systems was quite close, this post-evaluation was designed to get better insights on the differences of the systems and compare the CLIP-based text-query retrieval engines by introducing slight modifications to the original competition settings. An extended analysis of the overall results and the retrieval performance of all systems’ functionalities shows that a strong text retrieval model certainly helps, but has to be coupled with extensive browsing capabilities and other query-modalities to consistently solve known-item-search tasks in a large scale video database. 

Open Access

Journal article

International Journal of Multimedia Information Retrieval

Alejandro Moreo; Fabrizio Sebastiani; Pablo González

ISTI-CNR; University of Oviedo

Quantification is the supervised learning task that consists of training predictors of the class prevalence values of sets of unlabelled data, and is of special interest when the labelled data on which the predictor has been trained and the unlabelled data are not IID, i.e., suffer from dataset shift. To date, quantification methods have mostly been tested only on a special case of dataset shift, i.e., prior probability shift; the relationship between quantification and other types of dataset shift remains, by and large, unexplored. In this work we carry out an experimental analysis of how current quantification algorithms behave under different types of dataset shift, in order to identify limitations of current approaches and hopefully pave the way for the development of more broadly applicable methods. We do this by proposing a fine-grained taxonomy of types of dataset shift, by establishing protocols for the generation of datasets affected by these types of shift, and by testing existing quantification methods on the datasets thus generated. One finding that results from this investigation is that many existing quantification methods that had been found robust to prior probability shift are not necessarily robust to other types of dataset shift. A second finding is that no existing quantification method seems to be robust enough to dealing with all the types of dataset shift we simulate in our experiments. The code needed to reproduce all our experiments is publicly available at https:// github. com/pglez82/quant_ datasetshift

Open Access

Journal article

Data Mining and Knowledge Discovery

Christos Tzelepis; Georgios Tzimiropoulos; Ioannis Patras; Stella Bounareli; Vasileios Argyriou;

Kingston University London; Queen Mary University of London; University of Nottingham

In this paper, we present our framework for neural face/head reenactment whose goal is to transfer the 3D head orientation and expression of a target face to a source face. Previous methods focus on learning embedding networks for identity and head pose/expression disentanglement which proves to be a rather hard task, degrading the quality of the generated images. We take a different approach, bypassing the training of such networks, by using (fine-tuned) pre-trained GANs which have been shown capable of producing high-quality facial images. Because GANs are characterized by weak controllability, the core of our approach is a method to discover which directions in latent GAN space are responsible for controlling head pose and expression variations. We present a simple pipeline to learn such directions with the aid of a 3D shape model which, by construction, inherently captures disentangled directions for head pose, identity, and expression. Moreover, we show that by embedding real images in the GAN latent space, our method can be successfully used for the reenactment of real-world faces. Our method features several favorable properties including using a single source image (one-shot) and enabling cross-person reenactment. Extensive qualitative and quantitative results show that our approach typically produces reenacted faces of notably higher quality than those produced by state-of-the-art methods for the standard benchmarks of VoxCeleb1 & 2.

Open Access

Journal article

Ioannis Patras; Zheng Gao;

Queen Mary University of London;

Self-supervised pre-training has been proved to be effective in learning transferable representations that benefit various visual tasks. This paper asks this question: can self-supervised pre-training learn general facial representations for various facial analysis tasks? Recent efforts toward this goal are limited to treating each face image as a whole, i.e., learning consistent facial representations at the image-level, which overlooks the consistency of local facial representations (i.e., facial regions like eyes, nose, etc). In this work, we make a first attempt to propose a novel self-supervised facial representation learning framework to learn consistent global and local facial representations, Facial Region Awareness (FRA). Specifically, we explicitly enforce the consistency of facial regions by matching the local facial representations across views, which are extracted with learned heatmaps highlighting the facial regions. Inspired by the mask prediction in supervised semantic segmentation, we obtain the heatmaps via cosine similarity between the per-pixel projection of feature maps and facial mask embeddings computed from learnable positional embeddings, which leverage the attention mechanism to globally look up the facial image for facial regions. To learn such heatmaps, we formulate the learning of facial mask embeddings as a deep clustering problem by assigning the pixel features from the feature maps to them. The transfer learning results on facial classification and regression tasks show that our FRA outperforms previous pre-trained models and more importantly, using ResNet as the unified backbone for various tasks, our FRA achieves comparable or even better performance compared with SOTA methods in facial analysis tasks.

Open Access

Conference paper

IEEE Conference on Computer Vision and Pattern Recognition

Aleksandra Kuczerawy Lidia Dutklewicz; Noémie Krack Peggy Valcke

KU Leuven

This chapter discusses how AI technologies permeate the media sector. It sketches opportunities and benefits of the use of AI in media content gathering and production, in media content distribution, in fact-checking and content moderation. The chapter then zooms in on ethical and legal risks raised by AI-driven media applications: lack of data availability, poor data quality and bias in training datasets, lack of transparency, risks for the right to freedom of expression, threats to media freedom and pluralism online, and threats to media independence. Finally, the
chapter introduces the relevant elements of the EU legal framework which aim to mitigate these risks, such as the Digital Services Act, the European Media Freedom Act proposal and the AI Act proposal.

Open Access

Book section

Cambridge handbook on the law, ethics and policy of Artificial Intelligence

Konstantinos Gkrispanis; Nikolaos Gkalelis; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas

Open Access

Conference paper

Winter Conference on Applications of Computer Vision

Evlampios Apostolidis; Konstantinos Apostolidis; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas

Open Access

Conference paper

Conference on Multimedia Modeling

Evlampios Apostolidis; Ioannis Kontostathis; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas

Open Access

Conference paper

Conference on Multimedia Modeling

Bogdan Ionescu; Hannes Fassold; Mihai Dogariu Werner Bailer;

Joanneum Research; University Politehnica of Bucharest

Open Access

Conference paper

Multimedia Modeling Conference

Claudio Gennaro; Claudio Vairo; Fabio Carrara; Fabrizio Falchi; Giuseppe Amato; Lucia Vadicamo; Nicola Messina; Paolo Bolettieri;

ISTI-CNR;

In this paper, we introduce the fifth release of VISIONE, an advanced video retrieval system offering diverse search functionalities. The user can search for a target video using textual prompts, drawing objects and colors appearing in the target scenes in a canvas, or images as query examples to search for video keyframes with similar content.
Compared to the previous version of our system, which was runner-up at VBS 2023, the forthcoming release, set to participate in VBS 2024, showcases a refined user interface that enhances its usability and updated AI models for more effective video content analysis.

Open Access

Conference paper

N/A

Alberto Del Bimbo; Federico Becattini; Francesco Marchetti; Lorenzo Seidenari;

Università degli Studi di Firenze; University of Florence;

Effective modeling of human interactions is of utmost importance when forecasting behaviors such as future trajectories. Each individual, with its motion, influences surrounding agents since everyone obeys to social non-written rules such as collision avoidance or group following. In this paper we model such interactions, which constantly evolve through time, by looking at the problem from an algorithmic point of view, i.e., as a data manipulation task. We present a neural network based on an end-to-end trainable working memory, which acts as an external storage where information about each agent can be continuously written, updated and recalled. We show that our method is capable of learning explainable cause-effect relationships between motions of different agents, obtaining state-of-the-art results on multiple trajectory forecasting datasets.

Open Access

Journal article

IEEE Transaction on Pattern Analysis and Machine Intelligence

Andrea Esuli; Davide Alessandro Coccomini; Fabrizio Falchi; Nicola Messina;

Institute of Information Science and Technologies

With the increasing importance of multimedia and multilingual data in online encyclopedias, novel methods are needed to fill domain gaps and automatically connect different modalities for increased accessibility. For example, Wikipedia is composed of millions of pages written in multiple languages. Images, when present, often lack textual context, thus remaining conceptually floating and harder to find and manage.
In this work, we tackle the novel task of associating images from Wikipedia pages with the correct caption among a large pool of available ones written in multiple languages, as required by the image-caption matching Kaggle challenge organized by the Wikimedia Foundation. A system able to perform this task would improve the accessibility and completeness of the underlying multi-modal knowledge graph in online encyclopedias. We propose a cascade of two models powered by the recent Transformer networks able to efficiently and effectively infer a relevance score between the query image data and the captions. We verify through extensive experiments that the proposed cascaded approach effectively handles a large pool of images and captions while maintaining bounded the overall computational complexity at inference time.
With respect to other approaches in the challenge leaderboard, we can achieve remarkable improvements over the previous proposals (+8% in nDCG@5 with respect to the sixth position) with constrained resources.

Open Access

Journal article

Multimedia Tools and Applications

Adrian Popescu; Bertrand Delezoide Céline Hudelot; David Picard; Eva Feillet; Grégoire Petit; Michael Soumm

CEA; Université Gustave Eiffel; Université Paris-Saclay;

Class-Incremental Learning (CIL) aims to build classification models from data streams. At each step of the CIL process, new classes must be integrated into the model. Due to catastrophic forgetting, CIL is particularly challenging when examples from past classes cannot be stored, the case on which we focus here. To date, most approaches are based exclusively on the target dataset of the CIL process. However, the use of models pre-trained in a self-supervised way on large amounts of data has recently gained momentum.
The initial model of the CIL process may only use the first batch of the target dataset, or also use pre-trained weights obtained on an auxiliary dataset. The choice between these two initial learning strategies can significantly influence the performance of the incremental learning model, but has not yet been studied in depth. Performance is also influenced by the choice of the CIL algorithm, the neural architecture, the nature of the target task, the distribution of classes in the stream and the number of examples available for learning.
We conduct a comprehensive experimental study to assess the roles of these factors. We present a statistical analysis framework that quantifies the relative contribution of each
factor to incremental performance. Our main finding is that the initial training strategy is the dominant factor influencing the average incremental accuracy, but that the choice of
CIL algorithm is more important in preventing forgetting.
Based on this analysis, we propose practical recommendations for choosing the right initial training strategy for a given incremental learning use case. These recommendations are intended to facilitate the practical deployment of incremental learning.

Open Access

Conference paper

N/A

Alberto Del Bimbo; Davide Pucci Federico Becattini;

Università degli Studi di Firenze; University of Florence;

Action understanding is a fundamental computer vision branch for several applications, ranging from surveillance to robotics. Most works deal with localizing and recognizing the action in both time and space, without providing a characterization of its evolution. Recent works have addressed the prediction of action progress, which is an estimate of how far the action has advanced as it is performed. In this paper, we propose to predict action progress using a different modality compared to previous methods: body joints. Human body joints carry very precise information about human poses, which we believe are a much more lightweight and effective way of characterizing actions and therefore their execution. Estimating action progress can in fact be determined based on the understanding of how key poses follow each other during the development of an activity. We show how an action progress prediction model can exploit body joints and integrate it with modules providing keypoint and action information in order to be run directly from raw pixels. The proposed method is experimentally validated on the Penn Action Dataset.

Open Access

Journal article

MDPI

Alberto Messina; Angelo Bruccoleri; Fulvio Negro; Maurizio Montagnuolo; Roberto Iacoviello;

RAI;

Open Access

Conference paper

International Conference on Big Data

Alberto Del Bimbo; Andrea Ciamarra Federico Becattini; Lorenzo Seidenari; Roberto Cardelli;

Mercatorum University; University of Florence;

The ever-increasing use of synthetically generated content in different sectors of our everyday life, one for all media information, poses a strong need for deepfake detection tools in order to avoid the proliferation of altered messages. The process to identify manipulated content, in particular images and videos, is basically performed by looking for the presence of some inconsistencies and/or anomalies specifically due to the fake generation process. Different techniques exist in the scientific literature that exploit diverse ad-hoc features in order to highlight possible modifications. In this paper, we propose to investigate how deepfake creation can impact on the characteristics that the whole scene had at the time of the acquisition. In particular, when an image (video) is captured the overall geometry of the scene (e.g. surfaces) and the acquisition process (e.g. illumination) determine a univocal environment that is directly represented by the image pixel values; all these intrinsic relations are possibly changed by the deepfake generation process. By resorting to the analysis of the characteristics of the surfaces depicted in the image it is possible to obtain a descriptor usable to train a CNN for deepfake detection: we refer to such an approach as SurFake. Experimental results carried out on the FF + + dataset for different kinds of deep-fake forgeries and diverse deep learning models confirm that such a feature can be adopted to discriminate between pristine and altered images; furthermore, experiments witness that it can also be combined with visual data to provide a certain improvement in terms of detection accuracy.

Open Access

Conference paper

N/A

Alberto Del Bimbo; Federico Becattini; Lorenzo Seidenari; Luca Cultrera; Pietro Pala

Università degli Studi di Firenze; University of Florence;

Conditional Imitation learning is a common and effective approach to train autonomous driving agents. However, two issues limit the full potential of this approach: (i) the inertia problem, a special case of causal confusion where the agent mistakenly correlates low speed with no acceleration, and (ii) low correlation between offline and online performance due to the accumulation of small errors that brings the agent in a previously unseen state. Both issues are critical for state-aware models, yet informing the driving agent of its internal state as well as the state of the environment is of crucial importance. In this article we propose a multi-task learning agent based on a multi-stage vision transformer with state token propagation. We feed the state of the vehicle along with the representation of the environment as a special token of the transformer and propagate it throughout the network. This allows us to tackle the aforementioned issues from different angles: guiding the driving policy with learned stop/go information, performing data augmentation directly on the state of the vehicle and visually explaining the model’s decisions. We report a drastic decrease in inertia and a high correlation between offline and online metrics.

Open Access

Journal article

IEEE Transactions on Intelligent Vehicles

Claudio Gennaro; Claudio Vairo; Fabio Carrara; Fabrizio Falchi; Giuseppe Amato; Lucia Vadicamo; Nicola Messina; Paolo Bolettieri;

ISTI-CNR;

This paper presents a revised version of the VISIONE video retrieval system, which offers a wide range of search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search.  The system is designed to ensure scalability using advanced indexing techniques and effectiveness using cutting-edge Artificial Intelligence technology for visual content analysis. VISIONE was the runner-up in the 2023 Video Browser Showdown competition, demonstrating its comprehensive video retrieval capabilities. In this paper, we detail the improvements made to the search and browsing interface to enhance its usability for non-expert users. 
A demonstration video of our system with the restyled interface, showcasing its capabilities on over 2,300 hours of diverse video content, is available online at {\url{https://youtu.be/srD3TCUkMSg}}. 

Open Access

Conference paper

Conference on Content-based Multimedia Indexing

Andrei Cosmin Jitaru Bogdan Ionescu; Mihai Dogariu Mihai Gabriel Constantin

University Politehnica of Bucharest

Memorability is a critical aspect of human cognition that has been studied extensively in various fields, including psychology, education, and computer vision. The ability to remember information and experiences over time is essential for learning, decision-making, and creating lasting impressions. While the number of computer vision works that attempt to predict the memorability score of videos has recently seen a significant boost, thanks to several benchmarking tasks and datasets, some questions related to the performance of automated systems on certain types of videos are still largely unexplored. Given this, we are interested in discerning what makes a video sample easy or hard to classify or predict from a memorability standpoint. In this paper, we use a large set of runs, created and submitted by the participants to the MediaEval Predicting Video Memorability task, and, using their results and a set of visual, object, and annotator-based features and analyses, we attempt to find and define common traits that make the memorability scores of videos hard or easy to predict.

Open Access

Conference paper

Conference on Content-based Multimedia Indexing

Ambrish Rawat; Anisa Halimi; Nathalie Baracaldo; Swanand Kadhe;

IBM Research;

Open Access

Conference paper

Socially Responsible Language Modelling Research

Elena Cabrio Mariana Chaves Pierpaolo Goffredo Serena Villata

CNRS Inria; Université Côte d'Azur;

Fallacies are arguments that employ faulty reasoning. Given their persuasive and seemingly valid nature, fallacious arguments are often used in political debates. Employing these misleading arguments in politics can have detrimental consequences for society, since they can lead to inaccurate conclusions and invalid inferences from the public opinion and the policymakers. Automatically detecting and classifying fallacious arguments represents therefore a crucial challenge to limit the spread of misleading or manipulative claims and promote a more informed and healthier political discourse. Our contribution to address this challenging task is twofold. First, we extend the ElecDeb60To16 dataset of U.S. presidential debates annotated with fallacious arguments, by incorporating the most recent Trump-Biden presidential debate. We include updated token level annotations, incorporating argumentative components (i.e., claims and premises), the relations between these components (i.e., support and attack), and six categories of fallacious arguments (i.e., Ad Hominem, Appeal to Authority, Appeal to Emotion, False Cause, Slippery Slope, and Slogans). Second, we perform the twofold task of fallacious argument detection and classification by defining neural network architectures based on Transformers models, combining text, argumentative features, and engineered features. Our results show the advantages of complementing transformer-generated text representations with non-textual features.

Open Access

Conference paper

Association for Computational Linguistics Empirical Methods in Natural Language Processing

Luca Cuccovillo; Milica Gerhardt; Patrick Aichroth;

Fraunhofer IDMT;

In this study we propose a novel approach to audio phylogeny, i.e. the detection of relationships and transformations within a set of near-duplicate audio items, by leveraging a deep neural network for efficiency and extensibility. Unlike existing methods, our approach detects transformations between nodes in one step, and the transformation set can be expanded by retraining the neural network without excessive computational costs. We evaluated our method against the state of the art using a self-created and publicly released dataset, observing a superior performance in reconstructing phylogenetic trees and heightened transformation detection accuracy. Moreover, the ability to detect a wide range of transformations and to extend the transformation set make the approach suitable for various applications.

Open Access

Conference paper

IEEE International Workshop of Information Forensics and Security

Christina Katsini; George E. Raptis; Vasilis Theodorou;

Human Opsis

Open Access

Conference paper

International Conference on Computer and Applications

Angelo Canale; Fabrizio Falchi; Giovanni Benelli; Giuseppe Amato; Luca Ciampi; Luca Incrocci; Stefano Chessa; Valeria Zeni;

ISTI-CNR; University of Pisa

Open Access

Journal article

Ecological Informatics

Hervé Le Borgne; Michel Crucianu; Nicolas Audebert; Perla Doubinsky

CEA; Conservatoire National des Arts et Métiers;

Open Access

Conference paper

Winter Conference on Applications of Computer Vision

Patrick Aichroth; Thomas Köllmer Zühal Kurt

Atilim University Fraunhofer IDMT;

The paper outlines an explainable knowledge graph-based recommendation system that aims to provide personalized news recommendations and tries to explain why an item is recommended to a particular user. The system leverages a knowledge graph (KG) that models the relationships between items and users’ preferences, as well as external knowledge sources such as item features and user profiles. The main objectives of this study are to train a recommendation model that can predict whether a user will click on a news article or not, and then obtain the explainable recommendations for the same purpose. This is achieved with three steps: Firstly, KG of the MIND dataset are generated based on the history and, the clicked information of the users, the category and subcategory of the news. Then, the path reasoning approaches are utilized to reach explainable paths of recommended news/items. Thirdly, the proposed KG-based model is evaluated using MIND News data sets. Experiments have been conducted using the MIND-demo and MINDsmall datasets, which are the open-source English news datasets for public research scope. Experimental results indicate that the proposed approach performs better in terms of recommendation explainability, making it a promising basis for developing transparent and interpretable recommendation systems.

Open Access

Conference paper Publication

Conference on Knowledge Discovery

Alberto Messina; Stefano Scotta;

RAI;

In this work, we present an example of how a relatively small Large Language Model (LLM) fine-tuned  to perform a simple and well defined task (assigning titles to news articles) could perform similarly or  even better than huge LLMs which are created to respond to any question. This approach of specializing  smaller LLMsonsimplertasksisalsointeresting because it goes in the direction of makingthis technology  more sustainable and available to a higher number of entities that usually could not use these expensive  models, both for economic and data policy reasons. We also present a couple of examples of how can be  evaluated the performances of LLMs when the task is specified as in the example that we present in this work.

Open Access

Conference paper

International Conference of the Italian Association for Artificial Intelligence

Albert Gatt Andrea Pedrotti; Anette Frank Aykut Erdem Emre Can Acikgoz Erkut Erdem Iacer Calixto Ilker Kesen Leticia Parcalabescu Michele Cafagna Mustafa Dogan

Hacettepe University Heidelberg University Institute of Information Science and Technologies Koç University University of Amsterdam; University of Malta University of Pisa Utrecht University;

With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs’ grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.

Open Access

Conference paper

Alberto Del Bimbo; Hondamunige Prasanna Silva Lorenzo Seidenari;

University of Florence;

This paper presents a novel reconstruction method that leverages Diffusion Models to protect machine learning classifiers against adversarial attacks, all without requiring any modifications to the classifiers themselves. The susceptibility of machine learning models to minor input perturbations renders them vulnerable to adversarial attacks. While diffusion-based methods are typically disregarded for adversarial defense due to their slow reverse process, this paper demonstrates that our proposed method offers robustness against adversarial threats while preserving clean accuracy, speed, and plug-and-play compatibility. Code at: https://github.com/HondamunigePrasannaSilva/DiffDefence.

Open Access

Conference paper

N/A

Marius Gavrilescu;

Technical University of Iasi

Open Access

Conference paper

E-Health and Bioengineering Conference 2023

Evlampios Apostolidis; Ioannis Patras; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas Queen Mary University of London;

Open Access

Conference paper

ACM Multimedia

Alberto Del Bimbo; Lorenzo Berlincioni; Marco Bertini; Stefano Berretti

Università degli Studi di Firenze; University of Florence;

Time-varying sequences of 3D point clouds, or 4D point clouds, are now being acquired at an increasing pace in several applications (personal avatar representation, LiDAR in autonomous or assisted driving). In many cases, such volume of data is transmitted, thus requiring that proper compression tools are applied to either reduce the resolution or the bandwidth. In this paper, we propose a new solution for upscaling and restoration of time-varying 3D video point clouds after they have been heavily compressed. Our model consists of a specifically designed Graph Convolutional Network that combines Dynamic Edge Convolution and Graph Attention Networks for feature aggregation in a Generative Adversarial setting. We present a different way to sample dense point clouds with the intent to make these modules work in synergy to provide each node with enough features about its neighbourhood in order to later on generate new vertices. Compared to other solutions in the literature that address the same task, our proposed model is capable of obtaining comparable results in terms of quality of the reconstruction, while using a substantially lower number of parameters (\simeq 300KB), making our solution deployable in edge computing devices.

Open Access

Conference paper

N/A

Artem Yaroshchuk; Christoforos Papastergiopoulos Dimitrios Tzovaras; Konstantinos Votis; Luca Cuccovillo; Patrick Aichroth;

CERTH - Center for Research and Technology Hellas Fraunhofer IDMT;

Open Access

Conference paper

IEEE International Workshop of Information Forensics and Security

Claudio Gennaro; Fabio Carrara; Giuseppe Amato; Lucia Vadicamo;

ISTI-CNR;

Open Access

Conference paper

International Conference on Similarity Search and Applications

Evlampios Apostolidis; Georgios Balaouras; Ioannis Patras; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas Queen Mary University of London;

Open Access

Book section

Encyclopedia of Information Science and Technology

Daniel Gatica-Perez; Sina Sajadmanesh

Idiap Research Institute

Graph Neural Networks (GNNs) have become a popular tool for learning on graphs, but their widespread use raised privacy concerns as graph data can contain personal or sensitive information. Differentially private GNN models have been recently proposed to preserve privacy while still allowing for effective learning over graph-structured datasets. however, achieving an ideal balance between accuracy and privacy in GNNs remains challenging due to the intrinsic structural connectivity of graphs. in this paper, we propose a new differentially private GNN called ProGAP that uses a progressive training scheme to improve such accuracy-privacy trade-offs. Combined with the aggregation perturbation technique to ensure differential privacy, ProGAP splits a GNN into a sequence of overlapping submodels that are trained progressively, expanding from the first submodel to the complete model. Specifically, each submodel is trained over the privately aggregated node embeddings learned and cached by the previous submodels, leading to an increased expressive power compared to previous approaches while limiting the incurred privacy costs. We formally prove that ProGAP ensures edge-level and node-level privacy guarantees for both training and inference stages, and evaluate its performance on benchmark graph datasets. Experimental results demonstrate that ProGAP can achieve up to 5-10% higher accuracy than existing state-of-the-art differentially private GNNs. Our code is available at https://github.com/sisaman/ProGAP.

Open Access

Publication

N/A

Alejandro Moreo; Fabrizio Sebastiani; Martin Senz; Mirko Bunse;

ISTI-CNR; University of Dortmund;

Quantification, i.e., the task of training predictors of the class prevalence values in sets of unlabeled data items, has received increased attention in recent years. However, most quantification research has concentrated on developing algorithms for binary and multiclass problems in which the classes are not ordered. Here, we study the ordinal case, i.e., the case in which a total order is defined on the set of n > 2 classes. We give three main contributions to this field. First, we create and make available two datasets for ordinal quantification (OQ) research that overcome the inadequacies of the previously available ones. Second, we experimentally compare the most important OQ algorithms proposed in the literature so far. To this end, we bring together algorithms proposed by authors from very different research fields, such as data mining and astrophysics, who were unaware of each others’ developments. Third, we propose a novel class of regularized OQ algorithms, which outperforms existing algorithms in our experiments. The key to this gain in performance is that our regularization prevents ordinally implausible estimates, assuming that ordinal distributions tend to be smooth in practice. We informally verify this assumption for several real-world applications.

Open Access

Publication

Data Mining and Knowledge Discovery

Fabrizio Falchi; Jan Sedmidubsky; Nicola Messina; Tomás Rebok;

ISTI-CNR; Masaryk University;

Open Access

Conference paper

N/A

Florin Leon; Marius Gavrilescu; Sabina-Adriana Floria;

Technical University of Iasi

Open Access

Conference paper

International Conference on System Theory

Anastasios Gkagkas; Davide Alessandro Coccomini; Gylfi Þór Guðmundsson; Jakub Lokoč; Jiaxin Wu; Nick Pantelidis; Nicola Messina; Rahel Arnold; Silvan Heller; Vera Benz; Werner Bailer;

CERTH - Center for Research and Technology Hellas Charles University; City University Hong Kong; ISTI-CNR; Joanneum Research; Reykjavik University; University of Basel;

Open Access

Conference paper

Conference on Multimedia Retrieval

David Renaudie; Georgios N. Yannakakis; Konstantinos Makantasis; Kosmas Pinitas; Matthew Barthet Mike Thomsen;

Massive Entertainment Ubisoft University of Malta

Open Access

Paper

Conference on Multimodal Interaction

Evlampios Apostolidis; Georgios Balaouras; Ioannis Patras; Vasileios Mezaris

CERTH - Center for Research and Technology Hellas Queen Mary University of London;

This paper presents a new reinforcement-based method for video thumbnail selection (called RL-DiVTS), that relies on estimates of the aesthetic quality, representativeness and visual diversity of a small set of selected frames, made with the help of tailored reward functions. The proposed method integrates a novel diversity-aware Frame Picking mechanism that performs a sequential frame selection and applies a reweighting process to demote frames that are visually-similar to the already selected ones. Experiments on two benchmark datasets (OVP and YouTube), using the top-3 matching evaluation protocol, show the competitiveness of RL-DiVTS against other SoA video thumbnail selection and summarization approaches from the literature.

Open Access

Paper

IEEE International Conference on Image Processing

Alberto Del Bimbo; Lorenzo Seidenari; Luca Cultrera;

University of Florence;

Out-of-Distribution (OOD) detection is a crucial challenge in computer vision, especially when deploying machine learning models in the real world. In this paper, we propose a novel OOD detection method leveraging Visual Attention Heatmaps from a Vision Transformer (ViT) classifier. Our approach involves training a Convolutional Autoencoder to reconstruct attention heatmaps produced by a ViT classifier, enabling accurate image reconstruction and effective OOD detection. Moreover, our method does not require additional labels during training, ensuring efficiency and ease of implementation. We validate our approach on a standard OOD benchmark using CIFAR10 and CIFAR100. To test OOD in a real-world setting we also collected a novel dataset: WildCapture. Our new dataset comprises more than 60k wild animal shots, from 15 different wildlife species, taken via phototraps in varying lighting conditions. The dataset is fully annotated with animal bounding boxes and species.

Open Access

Conference paper

N/A

Cristian-Nicolae Butincu; Florin Leon; Lavinia-Eugenia Ferariu; Marius Gavrilescu;

Technical University of Iasi

Open Access

Report

N/A

Fabio Carrara; Fabrizio Falchi; Maurizio Tesconi;

ISTI-CNR; University of Pisa

Trends and opinion mining in social media increasingly focus on novel interactions involving visual media, like images and short videos, in addition to text.

In this work, we tackle the problem of visual sentiment analysis of social media images — specifically, the prediction of image sentiment polarity. While previous work relied on manually labeled training sets, we propose an automated approach for building sentiment polarity classifiers based on a cross-modal distillation paradigm; starting from scraped multimodal (text + images) data, we train a student model on the visual modality based on the outputs of a textual teacher model that analyses the sentiment of the corresponding textual modality.

We applied our method to randomly collected images crawled from Twitter over three months and produced, after automatic cleaning, a weakly-labeled dataset of ∼1.5 million images. Despite exploiting noisy labeled samples, our training pipeline produces classifiers showing strong generalization capabilities and outperforming the current state of the art on five manually labeled benchmarks for image sentiment polarity prediction.

Open Access

Publication

ECAI - European Conference on Artificial Intelligence

Ali Najm; Antonios Liapis; Despina Michael-Grigoriou; Emmanouil Xylakis; Georgios N. Yannakakis;

Cyprus University of Technology; University of Malta

Open Access

Conference paper

N/A

Ioanna Valsamara; Ioannis Pitas;

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Andreas Sochopoulos; Evangelos Charalampakis; Ioannis Mademlis; Ioannis Pitas; Sotirios Papadopoulos

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Dimitrios Papaioannou; Ioannis Pitas; Vasileios Mygdalis

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Anestis Kaimakamadis; Ioannis Pitas;

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Ioanna Valsamara; Ioannis Mademlis; Ioannis Pitas;

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Emmanouil Krasanakis; Ioanna Valsamara; Ioannis Mademlis; Ioannis Pitas;

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Hervé Le Borgne; Michel Crucianu; Nicolas Audebert; Perla Doubinsky

CNAM; Université Paris-Saclay;

Open Access

Conference paper

N/A

Alejandro Moreo; Fabrizio Sebastiani; Mirko Bunse; Pablo González

Consiglio Nazionale delle Ricerche; University of Oviedo

Open Access

Book

N/A

Florin Leon; Marius Gavrilescu;

Technical University of Iasi

Open Access

Conference paper

International Conference on Environmental Engineering and Management

Ioannis Mademlis; Ioannis Pitas; Michail Kaseris

Aristotle University of Thessaloniki;

Open Access

Preprint

N/A

Axel Roebel; Lenny Renault; Rémi Mignot;

Sorbonne Université

Open Access

Journal article

Journal of the Audio Engineering Society

Hannes Fassold;

Joanneum Research;

Open Access

Conference paper

Conference on Video and Signal Processing

Claudio Gennaro; Fabrizio Falchi; Gaetano Emanuele Valenti; Giuseppe Amato; Luca Ciampi; Nicola Messina;

ISTI-CNR; University of Pisa

Open Access

Conference paper

Conference on Image Analysis and Processing

Antonino Furnari; Claudio Gennaro; Fabrizio Falchi; Giovanni Maria Farinella; Nicola Messina;

ISTI-CNR; University of Catania;

Open Access

Journal article

Conference on Image Analysis and Processing

Axel Roebel; Lenny Renault; Rémi Mignot;

Sorbonne Université

Open Access

Conference paper

International Conference on Digital Audio Effects

Bruno Lepri; Linchao Bao; Marco de Nadai; Nicu Sebe; Yahui Liu Yajing Chen;

FBK; Tencent AI Lab University of Trento;

Closed Access

Journal article

IEEE Transactions on Multimedia

Marius Gavrilescu;

Technical University of Iasi

Open Access

Report

N/A

Antonios Liapis; Chintan Triverdi; Emmanouil Xylakis; Georgios N. Yannakakis; Konstantinos Makantasis; Kosmas Pinitas; Matthew Barthet

University of Malta

Open Access

Conference paper

Conference on Affective Computing and Intelligent Interaction Workshops and Demos

Christina Katsini; George E. Raptis; Vasilis Theodorou;

Human Opsis

Open Access

Conference paper

Human-Computer Interaction

Ambrish Rawat; Gabriele Picco; Giulio Zizzo; Myles Foley; Taesung Lee; Yufang Hou;

IBM Research; Imperial College London;

Open Access

Conference paper

N/A

Ioannis Patras; Zengqun Zhao;

Queen Mary University of London;

Open Access

Conference paper

N/A

Aaron Duane; Cathal Gurrin; Florian Spiess; Jakub Lokoč; Klaus Schoeffmann; Konstantin Schall; Ladislav Peška; Loris Sauter; Luca Rossetto; Lucia Vadicamo; Nicola Messina; Omar Shahbaz Khan; Stefanos Vrochidis; Stelios Andreadis; Thao-Nhu Nguyen; Werner Bailer; Zhixin Ma;

CERTH - Center for Research and Technology Hellas Charles University; Dublin City University; HTW Berlin; ISTI-CNR; IT University of Copenhagen; Joanneum Research; Klagenfurt University; Singapore Management University; University of Basel; University of Copenhagen; University of Zurich;

This paper presents the findings of the eleventh Video Browser Showdown competition, where sixteen teams competed in known-item and ad-hoc search tasks. Many of the teams utilized state-of-the-art video retrieval approaches that demonstrated high effectiveness in challenging search scenarios. In the paper, a broad survey of all utilized approaches is presented in connection
with an analysis of the performance of participating teams. Specifically, both high-level performance indicators are presented with overall statistics as well as an in-depth analysis of the performance of selected tools implementing result set logging. The analysis reveals evidence that the CLIP model represents a versatile tool for cross-modal video retrieval when combined with interactive search capabilities. Furthermore, the analysis investigates the effect of different users and text query properties on the performance in search tasks. Last but not least, lessons learned from search task preparation are presented, and a new direction for adhoc search based tasks at Video Browser Showdown is introduced.

Open Access

Journal article

N/A

Ioannis Pitas; Vasileios Mygdalis

Aristotle University of Thessaloniki;

Open Access

Conference paper

N/A

Alexandros Zamichos; Ioannis Pitas; Vasileios Mygdalis

Aristotle University of Thessaloniki;

Open Access

Conference paper

N/A

Ioannis Pitas; Stefania Altini; Vasileios Mygdalis

Aristotle University of Thessaloniki;

Different adversarial attack methods have been proposed in the literature, mainly focusing on attack efficiency and visual quality, e.g., similarity with the non-adversarial examples. These properties enable the use of adversarial attacks for privacy protection against automated classification systems, while maintaining utility for human users. In this paradigm, when privacy restrictions are lifted, access to the original data should be restored, for all stakeholders. This paper addresses exactly this problem. Existing adversarial attack methods cannot reconstruct the original data from the adversarial ones, leading to significant storage overhead for all privacy applications. To solve this issue, we propose AdvRevGAN, a novel Neural Network architecture that generates reversible adversarial examples. We evaluate our approach in classification problems, where we examine the case where adversarial attacks are constructed by a neural network, while the original images are reconstructed using the reverse transformation from the adversarial examples. We show that adversarial attacks using this approach maintain and even increase their efficiency, while the classification accuracy of the model in the reconstructed data can almost totally be restored.

Open Access

Conference paper

N/A

Daniel Aláez; Ioannis Pitas; Jesús Villadangos; Vasileios Mygdalis

Aristotle University of Thessaloniki; University of Navarre;

Open Access

Conference paper

N/A

Christos Tzelepis; Georgios Tzimiropoulos; Ioannis Patras; Stella Bounareli; Vasileios Argyriou;

Kingston University London; Queen Mary University of London;

Open Access

Conference paper

N/A

Adrian Popescu; Armelle Brun; Evan Dufraisse; Jérôme Deshayes-Chossart; Julien Tourille;

Université de Lorraine; Université Paris-Saclay;

Target-dependent sentiment classification (TSC) enables a fine-grained automatic analysis of sentiments expressed in texts.
Sentiment expression varies depending on the domain, and it is necessary to create domain-specific datasets.
While socially important, TSC in the news domain remains relatively understudied.
We introduce MAD-TSC, the first multilingual aligned dataset designed for TSC in news. MAD-TSC differs substantially from existing resources.
First, it includes aligned examples in eight languages to facilitate a comparison of performance for individual languages, and a direct comparison of human and machine translation.
Second, the dataset is sampled from a diversified parallel news corpus, and is diversified in terms of news sources and geographic spread of entities.
Finally, MAD-TSC is more challenging than existing datasets because its samples are more complex.
We exemplify the use of MAD-TSC with comprehensive monolingual and multilingual experiments.
The latter shows that machine translations can successfully replace manual ones, and that performance for all included languages can match that of English by automatically translating test examples.

Open Access

Conference paper

Conference on Computational Linguistics N/A

Daniele Ugo Leonzio; Luca Cuccovillo; Marco Marcon; Paolo Bolettieri; Patrick Aichroth; Stefano Tubaro;

Fraunhofer IDMT; Politecnico di Milano;

In recent years, the multimedia forensic community has put a great effort in developing solutions to assess the integrity and authenticity of multimedia objects, focusing especially on manipulations applied by means of advanced deep learning techniques. However, in addition to complex forgeries as the deepfakes, very simple yet effective manipulation techniques not involving any use of state-of-the-art editing tools still exist and prove dangerous. This is the case of audio splicing for speech signals, i.e., to concatenate and combine multiple speech segments obtained from different recordings of a person in order to cast a new fake speech. Indeed, by simply adding a few words to an existing speech we can completely alter its meaning. In this work, we address the overlooked problem of detection and localization of audio splicing from different models of acquisition devices. Our goal is to determine whether an audio track under analysis is pristine, or it has been manipulated by splicing one or multiple segments obtained from different device models. Moreover, if a recording is detected as spliced, we identify where the modification has been introduced in the temporal dimension. The proposed method is based on a Convolutional Neural Network (CNN) that extracts model-specific features from the audio recording. After extracting the features, we determine whether there has been a manipulation through a clustering algorithm. Finally, we identify the point where the modification has been introduced through a distance-measuring technique. The proposed method allows to detect and localize multiple splicing points within a recording.

Open Access

Journal article

Multimedia FORensics in the WILD

Alberto Del Bimbo; Federico Becattini; Lorenzo Seidenari; Luca Cultrera;

University of Florence;

Open Access

Journal article

N/A

Nicu Sebe; Wei Wang; Yue Song

Beijing Jiaotong University; University of Trento;

Closed Access

Journal article

IEEE Transaction on Pattern Analysis and Machine Intelligence

Fang Li; Jing Wang; Jun Zhang; Wengjing Li; Zhongcheng Wu; Zhun Zhong

Chinese Academy of Science; University of Trento;

Closed Access

Journal article

Transactions on Intelligent Transportation Systems;

Andy Keller; Max Welling; Nicu Sebe; Yue Song

University of Amsterdam; University of Trento;

Open Access

Conference paper

International Conference on Machine Learning

Emmanouil Krasanakis; Ioannis Kompatsiaris; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

Open Access

Journal article

N/A

Bin Ren; Hao Tang; Nicu Sebe; Wei Wang; Xia Li; Yiming Wang;

Beijing Jiaotong University; ETH Zurich; FBK; University of Trento;

For semantic-guided cross-view image translation, it is crucial to learn where to sample pixels from the source view image and where to reallocate them guided by the target view semantic map, especially when there is little overlap or drastic view difference between the source and target images. Hence, one not only needs to encode the longrange dependencies among pixels in both the source view image and target view semantic map but also needs to translate these learned dependencies. To this end, we propose a novel generative adversarial network, PI-Trans, which mainly consists of a novel Parallel-ConvMLP module and an Implicit Transformation module at multiple semantic levels. Extensive experimental results show that PI-Trans achieves the best qualitative and quantitative performance by a large margin compared to the state-of-the-art methods on two challenging datasets. The source code is available at https://github.com/Amazingren/PI-Trans.

Open Access

Conference paper

N/A

Chen Feng; Ioannis Patras;

Queen Mary University of London;

Open Access

Conference paper

N/A

Christos Tzelepis; Giorgios Kordopatis-Zilos; Giorgios Tolias; Ioannis Kompatsiaris; Ioannis Patras; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; Queen Mary University of London;

Open Access

Conference paper

N/A

Chen Feng; Ioannis Patras;

Queen Mary University of London;

Open Access

Conference paper

N/A

Alberto Del Bimbo; Andrea Leonardo Chiara Albisani Federico Becattini; Lisa Cresti Lorenzo Berlincioni; Luca Cultrera; Sara Picchioni

Università degli Studi di Firenze; University of Florence;

Open Access

Conference paper

Computer Vision Foundation

Alessandrio Betti Frédéric Precioso Gabriele Ciravegna Kevin Mottin Marco Gori

Politecnico di Torino Université Côte d'Azur; University of Siena

The deployment of Deep Learning (DL) models is still precluded in those contexts where the amount of supervised data is limited.
To answer this issue, active learning strategies aim at minimizing the amount of labelled data required to train a DL model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. These techniques are theoretically sound, but an understanding of the selected samples based on
their content is not straightforward, further driving non-experts to consider DL as a black-box. For the first time, here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples. In our Knowledge-driven Active Learning (KAL) framework, rule-based knowledge is converted into logic constraints and their violation is checked as a natural guide for sample selection. We show that even simple relationships among data and output classes offer a way to spot predictions for which the model need supervision. We empirically show that KAL (i) outperforms many active learning strategies, particularly in those contexts where domain knowledge is rich, (ii) it discovers data distribution lying far from the initial training data, (iii) it ensures domain experts that the provided knowledge is acquired by the model, (iv) it is suitable for regression and object recognition tasks unlike uncertainty-based strategies, and (v) its computational demand is low.

Open Access

Conference paper

N/A

Fabrizio Falchi; Jan Sedmidubsky; Nicola Messina; Tomás Rebok;

ISTI-CNR; Masaryk University;

Open Access

Conference paper

N/A

Giorgios Kordopatis-Zilos; Ioannis Kompatsiaris; Pantelis Dogoulis; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas

Open Access

Conference paper

N/A

Adrian Popescu; Bogdan Ionescu; Giorgios Kordopatis-Zilos; Luca Cuccovillo; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; Fraunhofer IDMT; Université Paris-Saclay; University Politehnica of Bucharest

Open Access

Conference paper

Conference on Multimedia Retrieval

Adrian Popescu; Bogdan Ionescu; Giorgios Kordopatis-Zilos; Luca Cuccovillo; Symeon Papadopoulos

CERTH - Center for Research and Technology Hellas Czech Technical University in Prague; Fraunhofer IDMT; Université Paris-Saclay; University Politehnica of Bucharest

Front matter of the proceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation, held in Thessaloniki (Greece) on June 12th, 2023. The full proceedings are available online at https://doi.org/10.1145/3591106.

Open Access

Book section

International Workshop on Multimedia AI against Disinformation

Claudio Gennaro; Claudio Vairo; Fabio Carrara; Fabrizio Falchi; Giuseppe Amato; Lucia Vadicamo; Nicola Messina; Paolo Bolettieri;

ISTI-CNR;

Open Access

Conference paper

Conference on Multimedia Retrieval

Lucile Sassatelli; Quentin Guimard

Université Côte d'Azur;

Adaptive bitrate (ABR) algorithms are used in streaming media to adjust video or audio quality based on the viewer’s network conditions to provide a smooth playback experience. With the rise of virtual reality (VR) headsets, 360° video streaming is growing rapidly and requires efficient ABR strategies to also adapt the video quality to the user’s head position. However, research in this field is often difficult to compare due to a lack of reproducible simulations. To address this problem, we provide SMART360, a 360° streaming simulation environment to compare motion prediction and adaptive bitrates strategies.
We provide sample inputs and baseline algorithms along with the simulator, as well as examples of results and visualizations that can be obtained with SMART360. The code and data are made publicly available.

Open Access

Conference paper

ACM Multimedia Systems Conference

Juanjuan Weng; Nicu Sebe; Shaozi Li; Zhiming Luo; Zhun Zhong

University of Trento; Xiamen University

Open Access

Transactions on Information Forensics and Security

Hao Tang;

ETH Zurich; Tencent AI Lab University of Oregon;