07. malSET: An Automated Symbolic Execution Toolkit for Assisting Malware Classification Systems

Abstract

Traditional signature-based malware classification systems are unable to keep pace with the rapid expansion and sophistication of modern malware specimens. These systems commonly rely on the consistent influx of malware signatures into a centralized database of known malicious signatures, usually with some human interaction or curation involved. More modern forms of dynamic, behavioral-based categorization systems have been developed to account for new instances of unknown or polymorphic malware without the necessity for a consistently updated signature database or the need for time-consuming expert intervention. However, many of these automated classification systems are developed using machine learning-based technologies that require vast quantities of training data to construct models capable of successfully classifying malware samples into their respective families. Therefore, to optimize the use of training data, additional automated tools must be incorporated to reduce the threshold of training required to construct valid classifiers. Such tools must analyze malware samples for statistical trends that provide further insight into other methodologies for categorizing these specimens into their respective families to accomplish this goal. This research encompasses the development of an automated malware analysis tool for systematically identifying and extracting recurring behavioral traits in malware provided through well-known, publicly available datasets. Utilizing symbolic execution, this tool symbolically emulates the execution of malware samples to extract system calls and various other behavioral characteristics as input data for automated classification systems. Therefore, the resulting data generated from this tool can be integrated as additional training data for improving the efficacy of machine learning-based malware classification systems.