Intelligent Detection of Unknown DDoS Attacks through Open Set Recognition and Reciprocal Points Learning

Abstract

Distributed Denial of Service (DDoS) attacks continue to evolve in complexity and scale,
making traditional closed-set intrusion detection systems ineffective against previously
unseen attack patterns. Conventional machine learning models assume that all possible attack
classes are known during training, which limits their ability to detect novel or zero-day DDoS
attacks. To address this limitation, this study proposes an Open Set Recognition (OSR)
framework for unknown DDoS attack detection using Reciprocal Points Learning (RPL). The
proposed approach enhances the model’s ability to distinguish between known traffic patterns
and unseen malicious behaviors by learning compact decision boundaries and representative
reciprocal points in feature space. By integrating deep feature extraction with reciprocal
representation learning, the system effectively identifies unknown attack instances while
maintaining high classification accuracy for known classes. Experimental evaluation on
benchmark network intrusion datasets demonstrates improved detection performance, reduced
false positives, and strong generalization capability compared to traditional closed-set models.
The proposed framework provides a robust and scalable solution for next-generation
intelligent network security systems capable of handling evolving cyber threats