Toward signature extraction of Metasploit encoding algorithms using static analysis

Abstract

Shellcode is a code injected by the attackers to vulnerable software to gain access to the command prompt. The byte patterns of shellcodes help the intrusion detection systems to detect this type of shellcodes. To avoid detection, encoding algorithms is used by the attacker to encode the byte patterns. The detection of these encoded shellcodes is a challenging problem. To detect these encoded shellcodes, we perform a static analysis of encoding algorithms of Metasploit engine to extract the byte patterns (signature) of these algorithms. Then, we introduce a regular expression-based language called GtS to express these signatures. The experimental results show the effectiveness of our signatures in terms of accuracy and false positive rate. © 2018 Inderscience Enterprises Ltd.