Publication Details
Title: Prudent Practices for Designing Malware Experiments: Status Quo and Outlook
Author: C. Rossow, C. J. Dietrich, C. Kreibich, C. Grier, V. Paxson, N. Pohlmann, H. Bos, and M. van Steen
Bibliographic Information: Proceedings of the 33rd IEEE Symposium on Security and Privacy (S&P 2012), pp. 65-79, San Francisco, California
Date: May 2012
Research Area: Networking and Security
Type: Article in conference proceedings
PDF: http://www.icsi.berkeley.edu/pubs/networking/ICSI_prudentpracticesfor12.pdf
Overview:
Malware researchers rely on the observation of malicious code in execution to collect datasets for a wide array of experiments, including generation of detection models, study of longitudinal behavior, and validation of prior research. For such research to reflect prudent science, the work needs to address a number of concerns relating to the correct and representative use of the datasets, presentation of methodology in a fashion sufficiently transparent to enable reproducibility, and due consideration of the need not to harm others. In this paper we study the methodological rigor and prudence in 36 academic publications from 2006–2011 that rely on malware execution. 40% of these papers appeared in the 6 highest-ranked academic security conferences. We find frequent shortcomings, including problematic assumptions regarding the use of execution-driven datasets (25% of the papers), absence of description of security precautions taken during experiments (71% of the articles), and oftentimes insufficient description of the experimental setup. Deficiencies occur in top-tier venues and elsewhere alike, highlighting a need for the community to improve its handling of malware datasets. In the hope of aiding authors, reviewers, and readers, we frame guidelines regarding transparency, realism, correctness, and safety for collecting and using malware datasets.
Acknowledgements:
We thank our shepherd David Brumley for his support in finalizing this paper. We also thank all anonymous reviewers for their insightful comments. We thank all our anonymous malware sample feeds. Moreover, we thank Robin Sommer for his valuable discussion input. This work was supported by the Federal Ministry of Education and Research of Germany (Grant 01BY1110, MoBE), the EU “iCode” project (funded by the Prevention, Preparedness and Consequence Management of Terrorism and other Security-related Risks Programme of the European Commission DG for Home Affairs), the EU FP7-ICT-257007 SysSec project, the US National Science Foundation (Grant 0433702) and Office of Naval Research (Grant 20091976). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of the funders.
Bibliographic Reference:
C. Rossow, C. J. Dietrich, C. Kreibich, C. Grier, V. Paxson, N. Pohlmann, H. Bos, and M. van Steen. Prudent Practices for Designing Malware Experiments: Status Quo and Outlook. Proceedings of the 33rd IEEE Symposium on Security and Privacy (S&P 2012), pp. 65-79, San Francisco, California, May 2012
Author: C. Rossow, C. J. Dietrich, C. Kreibich, C. Grier, V. Paxson, N. Pohlmann, H. Bos, and M. van Steen
Bibliographic Information: Proceedings of the 33rd IEEE Symposium on Security and Privacy (S&P 2012), pp. 65-79, San Francisco, California
Date: May 2012
Research Area: Networking and Security
Type: Article in conference proceedings
PDF: http://www.icsi.berkeley.edu/pubs/networking/ICSI_prudentpracticesfor12.pdf
Overview:
Malware researchers rely on the observation of malicious code in execution to collect datasets for a wide array of experiments, including generation of detection models, study of longitudinal behavior, and validation of prior research. For such research to reflect prudent science, the work needs to address a number of concerns relating to the correct and representative use of the datasets, presentation of methodology in a fashion sufficiently transparent to enable reproducibility, and due consideration of the need not to harm others. In this paper we study the methodological rigor and prudence in 36 academic publications from 2006–2011 that rely on malware execution. 40% of these papers appeared in the 6 highest-ranked academic security conferences. We find frequent shortcomings, including problematic assumptions regarding the use of execution-driven datasets (25% of the papers), absence of description of security precautions taken during experiments (71% of the articles), and oftentimes insufficient description of the experimental setup. Deficiencies occur in top-tier venues and elsewhere alike, highlighting a need for the community to improve its handling of malware datasets. In the hope of aiding authors, reviewers, and readers, we frame guidelines regarding transparency, realism, correctness, and safety for collecting and using malware datasets.
Acknowledgements:
We thank our shepherd David Brumley for his support in finalizing this paper. We also thank all anonymous reviewers for their insightful comments. We thank all our anonymous malware sample feeds. Moreover, we thank Robin Sommer for his valuable discussion input. This work was supported by the Federal Ministry of Education and Research of Germany (Grant 01BY1110, MoBE), the EU “iCode” project (funded by the Prevention, Preparedness and Consequence Management of Terrorism and other Security-related Risks Programme of the European Commission DG for Home Affairs), the EU FP7-ICT-257007 SysSec project, the US National Science Foundation (Grant 0433702) and Office of Naval Research (Grant 20091976). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of the funders.
Bibliographic Reference:
C. Rossow, C. J. Dietrich, C. Kreibich, C. Grier, V. Paxson, N. Pohlmann, H. Bos, and M. van Steen. Prudent Practices for Designing Malware Experiments: Status Quo and Outlook. Proceedings of the 33rd IEEE Symposium on Security and Privacy (S&P 2012), pp. 65-79, San Francisco, California, May 2012