Adaptive Multi-Modal Transformer Decoders for Robust Speech and Scene Classification Using Audio– Visual Fusion

Abstract

Transformer-based architectures have also contributed much to speech recognition and classification systems, but putting most of the current methods on unimodal acoustic inputs, they may not be fully robust in noiseful reallife scenarios. In this paper, we develop an Adaptive MultiModal Transformer Decoder architecture merging two data modalities to produce scenarios: audio and visual, to achieve accuracy during classification and contextual information. The technique of Mel Frequency Cepstral Coefficients (MFCCs) and image features in the face of the scene are proposed by the system that extracts Mel Frequency Cepstral Coefficients from the audio signals and integrates them with the image features with modality-specific encoders. A cross-modal attention component of a transformer decoder learns relation between modalities in a dynamic fashion and selectively placing trustworthy inputs in a changing acoustic scenario. This framework has been trained and tested on a publicly accessible scene classification dataset which is paired-audio-image data. The experimental findings reveal that, multimodal fusion compares well with unimodal baselines with classification accuracy of over 99% when evaluated in conditions of control. The adaptive attention-based fusion mechanism provides greater robustness to noises and degraded signals as well as computation efficiency. This work offers empirical results that a reliability-conscience multimodal combination in the transformer decoders yields a significant boost in performance on speech-related and environmental classification tasks to form the basis of scalable next-generation multimodal AI systems.