2021 · 2021 IEEE International Systems Conference (SysCon)
Malware System Calls Detection Using Hybrid System
Evidence basis: full-text-reviewed · Review status: catalog-reviewed; paper-author approval pending
kernel-tracing system-tracing anomaly-detection machine-learning observability
malware detection system-call analysis LTTng Trace Compass LSTM Transformer multi-head attention PyTorch Linux VMs sequence classification precision recall unknown attacks
Core contribution: The paper combines LTTng system-call monitoring with recurrent and Transformer-style models for malware and anomalous system-behavior detection.
Problem and motivation
System-call anomaly detection must handle variable-length behavioral sequences, imbalanced anomaly data, and attacks that do not match known misuse patterns. Simple n-gram/count approaches can lose temporal behavior (pp. 1-2).
Method and contribution
LTTng collects Linux system calls; a sliding window and message queue feed Trace Compass for collection/inspection. Variable-length sequences are converted to normalized fixed-length count vectors and batched per process. The model uses an LSTM followed by Transformer encoder blocks with multi-head attention, feed-forward layers, residual connections, layer normalization, and a linear classifier (pp. 3-5).
Findings and evidence
The reported configuration achieves 92.6% precision and 93.8% recall; the table's strongest prior LSTM entry is 91.6%/92.2% (pp. 5-6). Experiments use multiple VMs on a host described as Intel Core i7 4 GHz, 8 CPUs, 64 GB RAM; the paper describes "45k normal and anomalous" samples but does not establish whether this is combined or per-class. A second client/server laboratory setup uses the top 28 processes and an 80/20 split (p. 5).
Limitations and future directions
Limitations: The evidence is laboratory/VM-based; production overhead, cross-OS behavior, and generalization to genuinely unseen malware families are not established. Count-vector preprocessing may discard within-sequence order; this is an analyst inference, not an author-stated limitation. The source reports dropout 0.8 for the model configuration but Transformer encoder dropout 0.1 (p. 5), an internal inconsistency.
Future work: Improve preprocessing and representation beyond plain count vectors; investigate GAN-based approaches for immunization/higher accuracy on unknown attacks; develop more informative anomaly representations (p. 7).
Sources and identifiers
- Published version published
- Public full-text page public_full_text
When to cite this paper
Cite this paper when your work uses or compares a hybrid LSTM-Transformer architecture that combines recurrent modeling of sparse system-call sequences with Transformer global-dependency modeling.
- For a hybrid LSTM-Transformer architecture that combines recurrent modeling of sparse system-call sequences with Transformer global-dependency modeling.
- For LTTng-based collection and normalized system-call count-vector preprocessing in malware/anomaly detection.
- For the reported 92.6% precision / 93.8% recall comparison against classical, LSTM, BiLSTM, and Inception-like LSTM baselines, with the VM/lab scope stated.
Citation
@inproceedings{ezzatiJivan2021malwaresystem,
author = {Yue Guan and Naser Ezzati-Jivan},
title = {Malware System Calls Detection Using Hybrid System},
year = {2021},
booktitle = {2021 IEEE International Systems Conference (SysCon)},
pages = {1-8},
publisher = {IEEE},
doi = {10.1109/SysCon48628.2021.9447094},
url = {https://doi.org/10.1109/SysCon48628.2021.9447094}
}Other citation formats for Word and reference managers
Guan, Y., & Ezzati-Jivan, N. (2021). Malware System Calls Detection Using Hybrid System. In 2021 IEEE International Systems Conference (SysCon) (pp. 1-8). https://doi.org/10.1109/SysCon48628.2021.9447094Y. Guan and N. Ezzati-Jivan, "Malware System Calls Detection Using Hybrid System," in 2021 IEEE International Systems Conference (SysCon), pp. 1-8, 2021, doi: 10.1109/SysCon48628.2021.9447094