IDS-NGNN: A SmartNIC-based Intrusion Detection System Based on Reduce and Merge Nested Graph Neural Networks

The growing complexity of network attacks has outpaced the capabilities of traditional intrusion detection systems (IDS), which often rely on flat data structures that fail to capture complex relationships within networks. To address this limitation, we propose IDS-NGNN, a novel IDS that integrates hardware-offload SmartNIC preprocessing with a nested graph neural network (NGNN) architecture. Unlike standard Graph Neural Networks (GNN), IDS-NGNN jointly captures local and global dependencies using a three-layer design: an internal GNN for host-level activity, a nested graph module for hierarchical aggregation, and an external GNN for inter-host communication. SmartNIC acceleration enables efficient real-time processing of large-scale graph-structured network data at the edge. We evaluate IDS-NGNN on six public IDS datasets, including CIC-IDS-2017, CSE-CIC-IDS-2018, and ToN-IoT. Experimental results demonstrate that IDS-NGNN achieves up to 95% accuracy and 92% F1-score, while maintaining efficiency suitable for real-time 100 Gbps deployments.