A Unified Cross-Modal Representation Framework for Analyzing Latent Harmful Patterns in Visual–Textual Media

Abstract

The rapid growth of social media has significantly increased the use of memes as a popular communication medium, with millions shared daily. While many memes are humorous, a substantial number contain implicit or explicit hate speech, posing serious challenges for content moderation. Detecting such content is complex due to the multimodal nature of memes, where meaning emerges from the interaction of visual and textual elements rather than either modality alone. Traditional systems rely on manual moderation or text-based analysis. Human moderation is slow, subjective, and incapable of handling large-scale data efficiently. Similarly, text-only models fail to capture visual context, sarcasm, symbolism, and implicit intent conveyed through images. These limitations lead to inaccurate classification, high false positives, and the inability to detect subtle or context-dependent hate speech, making such systems ineffective for real-time applications. To overcome these challenges, a multimodal deep learning framework is proposed that integrates both visual and textual analysis. Visual features are extracted using the Vision Transformer (ViT), while textual features are obtained through the eXtreme Language Model (XLNet), enabling a deeper understanding of semantic and contextual relationships. These features are fused into a unified representation and classified using advanced machine learning models, including the Sparse Linear Integer Model (SLIM), along with Logistic Regression Classifier (LRC), Decision Tree Classifier (DTC), and KNearest Neighbors (KNN) for comparative evaluation. The proposed system improves detection accuracy, reduces misclassification, and enhances contextual understanding. It supports scalable and real-time deployment, contributing to safer online platforms and advancing multimodal artificial intelligence research.