THEORY AND HYPOTHESES
Instructional Theory and Design
The contemporary model for cybersecurity instruction is based on a lecture and laboratory approach (TeachThought, 2019). To confirm, we surveyed the cybersecurity courses taught at fifty tier-1/R1 universities in the United States as listed in US News and World Report College Rankings, which indicated the wide use of a dialectical-contextual social constructivism method in which classroom lectures and team-based tasks are paired with laboratory exercises. All of the programs examined were in MIS, CIS, CS, or other IT program such as technology management. Students were all upper-class undergraduates with prerequisites in operating systems and networking. Laboratory exercises commonly used in these programs are Wireshark®, GNS3®, Cain/Abel®, Achilles®, IDS such as Zabbix® and Solar Winds® along with various cryptography labs such as PGP/GPG. By most accounts, this approach has been shown to be effective for rote knowledge (Arthur, et al., 2003). The ability to learn and practice has demonstrated knowledge acquisition benefits (Ferdig, 2006; TeachThought, 2019). Because this is the most common in-use best practice learning model, we assumed this approach for our baseline comparative. This baseline course, used in all sections, was predicated on education to drive behavioral change by incorporating the following features (Arthur et al., 2003; Conetta, 2019; Hoke, Reuter, Romeas, Montariol, Schnell, & Faubert 2017; Sitzmann & Weinhardt, 2017):
Materials must be targeted with participant learning characteristics in mind. Participants should have materials presented to them in a way that it is clear why poor cybersecurity practices will adversely affect their missions, allowing for different learner characteristics and cognitive styles. By contextualizing the security training materials, cybersecurity can become an important means to helping participants achieve their educational goals as well as fostering effective learning outcomes.
Materials must be experientially in context for the learner. Learning materials are not sufficient to change habituated behaviors unless they are incorporated into an environment or ecosystem in which the learner will actually apply the knowledge. The materials must present commonly used technologies that the learner will likely encounter in the field. The goal is to present enough material to drive meaningful behavioral change, but not so much that it is overwhelming. Importantly, it must consider that rare anomalous activities are hard for humans to detect (c.f. Hogan & Bell, 2009); and likewise, too much stimuli tend to be ignored as noise (Banks, 2007). Moreover, the instruction must also consider the Anderson (2000), Baldwin and Ford (1988) and Burke (1997) foundational understanding of learning/knowledge decay through scaffolding and continuous reinforcement.
Materials must be actionable. Corporate and governmental infrastructure such as transaction servers and power grids have both shared and unique characteristics. The approach must allow for the learning materials to drive learners toward simple but effective steps they can take immediately to improve the cybersecurity of all aspects of typical operations. These considerations include procedural knowledge as well as domain general and domain specific knowledge.
Simulation as Learning Augmentation
There is substantial anecdotal and some scientific evidence that simulations may augment procedural, declarative, and experiential cybersecurity knowledge and hence learning effectiveness (Jin, Tu, Kim, Heffron, & White, 2018; Veksler, et al., 2018). Popular simulations include Checkmarx® Codebashing, and Secure Code Warrior®. Unfortunately, few studies have systematically tested this proposition (Voskoboinicov &. Melnyk, 2018); however, there is strong theoretical justification to support it (e.g. Miranda, 2018). The few studies that have looked at various aspects of cybersecurity simulations on learning (e.g. Hendrix, Al-Sherbaz, & Bloom, 2016; Jalali, Siegel, & Madnick, 2019, Jin et al., 2018; Landers & Armstrong, 2017; Miranda, 2018; Voskoboiniov &. Melnyk, 2018) have provided partial insights into how simulations may be utilized to augment cybersecurity training. These studies, however, have not cut across learning modes to identify modal contributions to the learning outcomes.
Nevertheless, one way in which simulations are surmised to improve learning effectiveness is by motivating and engaging the learner, largely because they are animated with procedural challenges in a manner similar to a game -i.e. they are “gamified” (Reio & Wiswell, 2001). Beyond this, simulations facilitate learning effectiveness through reinforced encoding specificity, in which learners incorporate the situational environment along with the educational tasks (Trafton & Trickett, 2001).
Next, simulations have the ability to facilitate the connection of mental representations to the real-world environment (Miranda, 2018), which should improve performance and promote positive behavioral change relative to cybersecurity hygiene (Goode, Levy, Hovav, & Smith 2018; Veksler, et al., 2018). Simulations also allow for experimentation in a controlled environment, so that students can learn experientially (Veksler, et al., 2018). Moreover, they are surmised to enhance cognitive cueing and improve metacognitive awareness by prompting learners to reflect on their learning progress and allowing them to repeat material at critical junctures if needed (Arthur, Bennett, Edens, & Bell, 2003; Conetta, 2019). Therefore,
H1. Cybersecurity simulations will improve applied learning performance compared to conventional classroom/lab study alone.
Live Activity Event as Learning Augmentation
A live activity such as a “hackathon” (or sometimes, live-fire-activity) or “capture the flag event” goes beyond simulation by placing the learner in active real-world situation in which participants compete to try to compromise and defend/remediate systems (Leune & Petrilli, 2017; Sommestad & Hallberg, 2012). Where simulations allow for reinforcement and elaborative rehearsal, a live activity “puts knowledge to the test” (Hoke, Reuter, Romeas, Montariol, Schnell, & Faubert 2017; Sitzmann & Weinhardt, 2017). Participants learn the effectiveness of what they have learned by means of practical application and execution of what they know (Landers & Armstrong, 2017). In that sense, it is a reinforcing reciprocal learning process – it reinforces what works, and illuminates what does not work (Hoke, et al., 2017).
Finally, unlike simulations, which are sequential, live activities are non-sequential in nature (Kirschner & Paas, 2001; Retalis & Skordalakis, 2002) requiring acute situational awareness and optimal behavioral habituation to respond effectively “on the fly” (Torkzadeh & Van Dyke, 2002). This mode of learning is surmised link information to the activity, which augments knowledge scaffolding opportunities (Hoke, et al., 2017) and enhances the student’s ability to gather, organize, and integrate information in order to apply it (Landers & Armstrong, 2017). As a result:
H2. Live activities will improve applied learning performance compared to conventional classroom/lab study alone.
H3. Live activities will improve applied learning performance compared to conventional classroom/lab study combined with simulations.