INTRODUCTION
The state of the cybersecurity among democratic nations is grave, and there is little dispute regarding the need to significantly improve (Veksler, Buchler, Hoffman, Cassenti, & Sugrim, 2018). This need is present in many different verticals, but it is especially pressing within organizational and national critical infrastructure, where attackers are highly motivated and the consequences of failure may be catastrophic (Center for Strategic and International Studies, 2017).
Due to the importance of the threats, obtaining access to information on cybersecurity matters is not particularly difficult. Bookstores, universities, and the Internet are overflowing with good advice and best practices. However, countermeasures are often not put into practice until after a problem has been discovered. We suffer not from ignorance of knowing what to do, but from a seeming inability or unwillingness to put the knowledge into practice. In other words, there is a significant knowing-doing gap (Workman, Bommer & Straub, 2008). Consequently, cybersecurity training has become a focal point in both inculcation of new information as well as refreshing awareness.
However, the few empirical studies of the efficacy of various training methods and modes that have been conducted have yielded inconsistent findings (Arthur, Bennett, Edens, & Bell, 2003; Thatcher & Perrewé, 2002; Veksler, et al., 2018). Recent literature on the use of gamified simulations (e.g. Jalali, Siegel & Madnick, 2019; Jin, Tu, Kim, Heffron, & White, 2018) have suggested that highly targeted learn-practice simulations carefully crafted to address the needs of a particular audience may present an opportunity for improving cybersecurity behaviors (i.e. doing better), leading to tangible improvements in the cybersecurity stance (Arthur, et al., 2003; Rumeser & Emsley, 2018).
Beyond gamified simulations, there has been speculation that “live-fire” exercises such as hackathons and capture the flag events may further improve learner capabilities (Ernits et al., 2015). Moreover, a survey of the literature (e.g. Ernits et al., 2015; Hoffman, et al., 2005; Schepens, et al., 2002) shows both the need and the value of cybersecurity games and competitions that go beyond the typical cyber training exercises and simulations, yet there have been few if any systematic tests of these propositions to our knowledge. Such a study could prove informative to the cybersecurity training literature, as simulations and competitive games have been shown to be effective in other areas such as identifying exploitable flaws in cyber infrastructure (Pan, Teixeira, López & Palensky, 2017).
In addition, domain general studies on training effectiveness (e.g. Arthur, et al., 2003) have shown that learning occurs best when the training is targeted to a specific set of behaviors or skills, and are situated in context relevant to the learner, and are actionable. In other words, training and development that can be used immediately rather than merely instilling “head knowledge.” Given these findings, best pedagogical practice uses the present-test-practice-assess (PTPA) approach to facilitate optimal learning-doing behaviors (See Figure 1). The PTPA model has been a commonly recognized bestpractice pedagogy dating back to Dewey (1998) in which practice immediately follows topical instruction.
To further inform the body of cybersecurity literature, we conducted a systematic test of three modes of cybersecurity education (classroom training, simulations, live-fire exercises) compared to the traditional baseline instruction (traditional model), as well as examined the interactions on training efficacy using the PTPA approach. For this, we provided short topical instruction, followed by a short quiz on the topic. This was the baseline. The simulations replaced the midterm and final exams in the traditional class/lab instruction baseline with passing two simulation challenges presented by Codebashing® and Secure Code Warrior®. Our live fire activities included a Universitywide hackathon against the OWASP® Mutillidae™ in a controlled environment, and a CTF365® activity. CTF365 is an online capture the flag environment commonly used for capture-the-flag instruction and competitions. Our primary interest was to determine the contribution of each mode of learning on cybersecurity response to factor into training evaluation and benefit analysis.
Figure 1: PTPA Training/Learning Approach