EXECUTION

5. EXECUTION

5. EXECUTION

Most of JV 3.0 planning revolved around an in-person exercise, originally scheduled for April 28–30. When COVID-19 forced the cancellation of the in-person event, planners had to shift to a completely virtual format. The final execution events took place on September 22 for Charleston, SC, and September 24 for Savannah, GA.

This chapter outlines how the JV execution events were administered. It discusses the composition and responsibilities of participants, the White Cell, and data collectors; the ROC Drills; the execution of the main events; and the ACI’s utilization of technical platforms to host the virtual exercises. First, however, we discuss the impact of COVID-19 as well as the rapidly developed Jack Pandemus exercise.

5.1. Coronavirus Disease 2019 (COVID-19) / Jack Pandemus

The most important partner-participants in JV are, and always will be, the municipal-level individuals and organizations. In particular, the emergency managers of participating cities have the responsibility and existing relationships to bring together the best coalitions for their respective areas. When it became clear that COVID-19 would not only interrupt JV 3.0, but also dominate real-life local incident planning and response for the foreseeable future, the main research objectives of JV became less urgent for these key partners. As a result, the Planning Team had to decide whether to introduce the pandemic, a topic on everyone’s mind, into JV 3.0. Additionally, the cities of Charleston and Savannah face hurricane season each year during the summer and early fall, thus pushing back any possibility of rescheduling a JV event to late September. Such a delay risked a loss of momentum for JV and a loss of engagement with all partners and participants. Fearing that the pandemic would be a confounding variable and make studying cyberattack response more difficult, the Planning Team decided to produce and conduct a virtual event that incorporated the pandemic into the scenario and used the same techniques and platforms as JV. The purposes of the resulting exercise, dubbed “Jack Pandemus,” were to:

Address participant concerns about cyber incident response during the current pandemic crisis;
Reengage the coalition of partners and participants with a highly relevant exercise and generate fresh momentum toward a rescheduled JV 3.0; and
Execute a trial run with the technical platforms planned for JV use to familiarize planners, facilitators, and participants with the technologies and their capabilities.

5.1.1. Conduct of Jack Pandemus

The ACI, in partnership with FTI Consulting and NUARI, executed Jack Pandemus twice: once for Charleston on June 23, 2020, and once for Savannah on June 30, 2020. The two-hour virtual TTXs used the DECIDE® platform to play through a hypothetical scenario that included a cyberattack on the local natural gas company and a gas pipeline disruption directly impacting electrical power generation and healthcare delivery.

The Jack Pandemus scenario occurred in the context of an ongoing pandemic response. The scenario included government-ordered shutdowns, nonpharmaceutical interventions, personal protective equipment shortages, and protests. During these challenges, a cyber intrusion at a gas company caused an explosion at a natural gas relay. Already short on personnel and with its resources overtaxed, the local government was forced to request assistance through the county emergency operations center to the state’s emergency management division.

5.1.2. Jack Pandemus Summary Findings and Feedback

As with JV, the scenario was designed to emphasize the holistic, multisector nature of incident response. Participants gained a much greater appreciation of both their cross-sector dependencies and the dependencies of other organizations and were often surprised how cyberattacks could have ripple effects. The event highlighted numerous sector interdependencies among hospitals, local schools, local vendors, power companies, state emergency operations, and the defense coordinating element (DCE). As a result, participants realized the importance of establishing relationships prior to a crisis.

Regarding municipal readiness, the scenario highlighted the significant disconnect between the resources available and the resources that were needed. City and county personnel were concerned with how quickly municipal resources were being exhausted and sought any available state and federal resources for cyber incidents. The scenario events’ ambiguous cause—cyberattack or equipment malfunction—confused communications between participating organizations. Due to this ambiguity, participants noted the critical need for clearer legal authorities, well-defined response procedures, and priorities for resources allocation prior to a real crisis.

The consensus among participants was that Jack Pandemus achieved its objectives and was a resounding success. Participants appreciated the challenging nature of the scenario and the lessons learned; most wished the scenario could have been longer than 2 hours. The ACI conducted a followup webinar on July 19, 2020, to review necessary changes to the JV 3.0 exercise in response to the pandemic. The successful execution of Jack Pandemus allowed participants and administrators to give essential feedback on the virtual, distributed execution which the Planning Team incorporated into the JV 3.0 rehearsals and main event.

5.2. Event Design

JV 3.0 was originally designed to be an in-person 3-day event. Changing to a virtual, single-day event while ensuring maximum value from the interactions between participants proved to be a significant challenge. However, the initial groundwork done by the Planning Team meant that the final event was effective and engaging. At both events, over 95 percent of participants stayed through the entire exercise, and everyone who participated contributed something during the event. This section describes the final design and the decisions and circumstances that determined how JV 3.0 was conducted.

5.2.1. Virtual Execution

There are many reasons to prefer in-person drills and exercises, including the benefits of an LFX, enhanced opportunities for discussion, and increased relationship building in general. However, the success of Jack Pandemus showed an incident response exercise could meet objectives in a virtual, distributed setting, and feedback from Jack Pandemus showed how virtual tables (breakout rooms) could be used to encourage engagement. Technology platforms allowed for the effective dissemination of information, organized and facilitated discussions, participant feedback, and data collection. Although a virtual event necessitates controlled communications—using the “raise your hand” function in Microsoft Teams, for example—the format allows all conversations to effectively be captured by data collectors. Also, having participants distributed throughout various locations in real time simulated a real incident response scenario.

5.2.2. Technology Platforms

During the virtual events for each city, attendees used Microsoft Teams to participate in and move between plenary discussions at the main table and assigned, small-group tables designed to replicate actual interactions within organizations during incident response. Participants needed to log into two separate Microsoft Teams meetings—one for each of these tables. Participants were assigned tables based on their organization or responsibility and were instructed to remain at the assigned tables throughout the exercise. In addition to oral participation via Microsoft Teams, participants could hold side discussions in the Teams chat, and all of this data was analyzed by data collectors.

The DECIDE® platform, described in section A.2.4 of this report, allowed participants to visualize scenario information, send and receive communications, and answer survey questions. Participants saw three panes on the DECIDE® screen: a communication pane highlighted in green, an information pane highlighted in yellow, and an action pane highlighted in orange. Data collectors and event controllers used a DECIDE® chat line to log critical participant discussions and a separate DECIDE® chat channel to log meta observations.

5.2.3. White Cell

The JV 3.0 exercise was administered by an exercise team, or “White Cell” (see Figure 6, “Exercise Team [White Cell]”). White Cell personnel were assigned to several different roles:

The exercise lead was responsible for the overall conduct of the event, including planning and platforms, dealing with changes, and providing guidance to the White Cell members. The exercise lead also engaged with the participant organizations’ points of contact.
The facilitator lead guided event discussions at the main table to meet drill objectives and, as such, needed to thoroughly understand the event scenarios and participant organization responsibilities. This role required striking a delicate balance between guiding the discussions and allowing for freedom of thought and action. Specific responsibilities included coordinating the overall event with the exercise lead; managing the information injects, including any necessary changes to them; monitoring participant responses and stress levels; encouraging interorganizational engagement; identifying gaps in policy or process; assessing cities’ incident response preparedness; and overseeing the event controllers and DECIDE® platform personnel.
The event controllers support a particular table of players, serving as both the communication link for the players to the White Cell and facilitator for the players at their discussion tables. The event controllers acted as communication bridges between the individual tables and the White Cell, using the Microsoft Teams White Cell chat line to summarize decisions made by the discussion table and points of discussion that would be of interest to the larger group. The event controller also used the DECIDE® White Cell chat channel to record metaobservations for event improvement. Other responsibilities included maintaining the event schedule, supporting participant use of DECIDE® and Microsoft Teams, and providing a summary of the events of the turn if necessary.
The data collectors²¹ captured critical observations, primarily using the DECIDE® platform, but also Microsoft Teams for redundancy, as the scenario unfolded. Data collectors were assigned to specific sectors or groups of interest; they also filled gaps in coverage when necessary. Following the Jack Voltaic 3.0 Data Collector Guide, data collectors familiarized themselves with the DECIDE® platform to document details of participant reactions and interactions. They captured probing questions posed by participants and communication or relationship gaps and identified interdependencies among people or organizations. Data collectors focused on critical insights during incident response, including:
- » Information, communication, and operational gaps discovered;
- » Newly identified interdependencies between participants;
- » Gaps and interdependencies that affected or concerned an organization;
- » Newly formed relationships, groups, and structures created during each scenario turn;
- » Actions assigned sectors took to mitigate the impacts of the cyber incident and scenario injects;
- » Internal and external information-sharing mechanisms of assigned sectors;
- » Interactions, collaborations, and friction points between the public and private sectors as the scenario unfolded; and
- » New and/or existing thresholds for requesting additional support during response efforts.
DECIDE® controllers managed the DECIDE® experience of each event, including planning; building; deploying; operating; maintaining; and adjusting, if necessary, a functional environment to support the event. They troubleshot and addressed any technical issues with the platform and assisted event controllers during the event.

Figure 6: Exercise Team (White Cell)

5.2.4. Event Format

Each single-day event began at 8:00 a.m. and concluded at 4 p.m. Participants played the scenario over a series of turns, and each turn included three phases (see Figure 7, “Turn Phases”):

Phase 1—Assess (approximately 10 minutes): The Assess phase was focused on the individual participant and his or her interaction with DECIDE®. During this phase, participants received their sector-specific injects and had time to digest the incoming information. Participants were encouraged to use DECIDE® communication features to contact participants at other tables to facilitate their discussions.
Phase 2—Discuss (approximately 15 minutes): The Discuss phase allowed participants to discuss the current situation with the other participants at their discussion tables. For JV 3.0, there were eight discussion tables:
- » City table
- » County table
- » State table
- » Health and medical table
- » Port and port operations table
- » Energy table
- » Federal table
- » Private sector table

During this phase, the objective of the discussion was to determine: - » What injects were most relevant to the organizations/roles at this table? - » What existing plans applied to these issues? Were there issues that were not covered by a plan? - » What decisions would you make in responding to the inject? - » What actions would you take in responding to the inject? If you would not take an action, why not?

Players could request information from an agency or sector not participating in the event by reaching out through the DECIDE® platform or notifying the discussion table’s event controller. Upon conclusion of this phase, participants transitioned to the next phase.

Phase 3—Integrate (approximately 45 minutes): The final phase, Integrate, was communityfocused and leveraged Microsoft Teams for a facilitated discussion on how the various organizations responded to the events of the turn. This phase required all participants to interact with the facilitator lead at the main table, which included all participants and the White Cell. During this phase, the facilitator lead had participants share their findings from the previous two phases with the community through directed and open-ended questions. At the end of this phase, participants completed survey questions.

Figure 7: Turn Phases

During each phase, data collectors used the DECIDE® platform to record observations based on their assigned areas.

5.2.5. Rehearsal of Concept Drills

Four ROC Drills were held virtually: two for Charleston on August 18, 2020, and September 8, 2020, and two for Savannah on August 20, 2020, and September 10, 2020. The goals of the ROC Drills were to further familiarize participants with the tools and to refine processes planned for use in the JV 3.0 exercises. Because JV 3.0 was originally intended to be a 3-day event, the scenario developed for JV 3.0 was too long for a single day. Rather than lose any value for unplayed turns, the drills provided an opportunity to introduce the scenario and the first four turns of the scenario. Lastly, the drills allowed the data collection team to practice the procedures in the Jack Voltaic 3.0 Data Collector Guide prior to JV 3.0.

The ROC Drills followed the same basic structure with respect to technical platform, player and White Cell roles and responsibilities, and event format as the main event, except that each drill was a 4-hour event. The first drill for each city included turns 1–3, and the second drill included turns 3–4. See table 4 for the schedule of the second ROC Drill. This effectively familiarized players and White Cell personnel with the scenario and helped solidify the format of the main event. In every drill—and, to a lesser extent, the main event—there were log-in and other technical issues, so overlapping the last turn of the rehearsal with the first turn of the main event mitigated the effect of some participants struggling to join on time.

Table 4: Schedule of Second ROC Drill

5.3. Event

Using the lessons learned from the Jack Pandemus exercise and the ROC Drills, the JV 3.0 exercises were conducted virtually for Charleston on September 22, 2020, and for Savannah on September 24, 2020. Because all participants logged in using different browsers from different locations around the country, with some participants logging in at home and others logging in from behind government or corporate firewalls, there were some technical issues with respect to logging in and seeing information presented in DECIDE®. Technical issues affected fewer than 10 participants per event and were resolved for all participants within 30 minutes of commencement. Having rehearsed event execution during the ROC Drills, the ACI was able to begin and end all turns within 1 minute of the planned times. Overall, execution of both events was smooth and efficient after the early-morning technical issues had been resolved. The event schedule is shown in table 5.

Table 5: Event Schedule

5.3.1. Event Participants

In addition to the White Cell, many organizations with varying roles and responsibilities participated in JV 3.0: city management, city and county emergency management, port authorities, county school districts, fire and police departments, utilities, railway companies, the National Guard, several federal agencies, as well as others (see Table 1, “JV 3.0 Participants,” in section 4.6). Some of these organizations communicate and depend on each other’s services regularly, even daily, while others may never work together except in a crisis. These dependencies and interactions—or lack thereof—were a focus of the JV 3.0 exercises.

Participants were provided with and asked to abide by the Jack Voltaic 3.0 Player Handbook, which contained standards, guidelines, and instructions geared toward attainment of the event goals and objectives. To facilitate realistic participant responses to the scenario, the ACI established certain expectations. Participants were asked to accept the scenario events at face value, rather than questioning or fighting the “facts.” Participants were asked to represent their organizations or sectors and react—given their existing capabilities, resources, and plans—as if the scenario were an actual incident. They were asked to execute their organizations’ crisis action or incident response plans and to note any gaps in processes or procedures as well as identify necessary internal and external resources. They were also asked to identify the limits of their decision making and the decision making of superiors and subordinates. Participants were encouraged to stay engaged and use the exercise as a learning opportunity, voicing opinions, discussing options, and highlighting opportunities for improvement.

5.3.2. Group Interaction

Interactions between the White Cell and the participants were limited to the conversations led by the facilitator lead because any other conversations would not be properly captured, as required by the data collection and analysis plan. To protect the integrity and flow of the exercise, interaction between different participant groups was encouraged, but only if it was to coordinate or act in response to the scenario stimuli. Support staff were instructed to communicate only with the facilitator lead and exercise lead to ensure messages were coordinated and only coming from the ACI.

Prior to the start of JV 3.0, participants were directed to register for the event and set up a DECIDE® account. To start each exercise, the exercise lead welcomed participants, explained the goals of the event, and described how to use DECIDE® and Microsoft Teams. NUARI provided access to the DECIDE® exercise environment, and the White Cell and participants accessed both DECIDE® and the applicable Microsoft Teams meeting rooms. The facilitator lead explained the plan and schedule for the day, instructed participants to begin turn 4 by moving to their respective tables within Microsoft Teams, instructed the DECIDE® controllers to populate the turn injects in the DECIDE® platform, and set the time for all participants to return to the main table.

5.3.3. After Action Review

Upon completion of the exercise, the facilitator lead led an After Action Review discussion at the main table focused on overall thoughts about the day’s events. Specifically, discussion centered on whether the exercise generated a better understanding of the possible risks and threats arising from a cyber incident, the players in the environment and their roles, and what the path forward should be. The purpose of this final discussion was to achieve a holistic assessment of the exercise and obtain recommendations for moving forward. Participants were instructed not to replay each event or to blame or otherwise attribute issues to specific organizations or participants; rather, they were asked to provide lessons learned, identify specific problems or issues, and recommend improvements. Finally, participants were asked to provide After Action Review comments in DECIDE®.

The initial feedback on JV 3.0 was primarily positive. Most participants thought the exercise was implemented effectively, despite COVID-19 causing the ACI to truncate and modify the event. The planning meetings and drills leading up to the exercise were recognized as having been very helpful.

As virtual meetings become more commonplace, many organizations struggle to adapt. To be successful, JV 3.0 required interaction among many different organizations of various types, whether they were private, public, federal, state, or local. Fortunately, the extensive planning and practice that the ACI conducted prior to the event proved to be both constructive and worthwhile.

Additional feedback and lessons learned can be found in chapter 6, “Findings.”

5.4. Post-Event

In the weeks following the JV 3.0 exercise, the ACI planners hosted a series of out-briefings with partners and participating organizations. The initial feedback on JV 3.0 was primarily positive. After participants had had time to reflect, they provided additional insights on the usefulness of the exercise and its potential moving forward. Notes taken during these conversations contributed to the findings located in chapter 6 of this report.

5.5. Executive Out-Brief and Discussion

Due to the virtual execution of the event, the ACI converted DV Day to a 90-minute Executive OutBrief. Held virtually on September 30, 2020, the ACI’s intent for this event was to provide an effective forum for informing and engaging public and private senior executives about the outcomes and lessons learned from the JV exercise.

In addition to describing JV 3.0 and its participants, the ACI shared the following initial observations and corresponding strategic implications for planning, preparation, execution, and resources:

Although many organizations are effective at dealing with natural disasters, many are not as prepared for cyber or information attacks. The interdependencies among sectors result in risks being shared by all; thus, everyone should review assumptions and adjust cyber incident response plans to improve resilience against potential cascading effects. Through increased information sharing and maintaining cross-sector partnerships, cities and private industry can achieve improved resilience through a whole-of-community approach.
JV 3.0 did not directly impact telecommunications. Redundant communication channels should be developed and readied for degraded operations.
The GA Emergency Management and Homeland Security Agency and SLED were very effective at bringing all of the incident response issues together.
JV 3.0 highlighted the increased needs of states, cities, and the private sector for trained cybersecurity personnel and funded programs. Training opportunities should be increased, technology enablers leveraged, and repeatable frameworks developed.

Some highlights of the executive response included the following:

It was widely agreed that JV 3.0 was a unique and timely exercise.
Regarding the JV 3.0 scenario, one DV said it was appropriate to have cascading events rather than one catastrophic one because having numerous, smaller events forces players to identify thresholds for when to recognize purposeful threats.
Recognizing the value in developing and maintaining cross-sector and cross-jurisdiction relationships to encourage a whole-of-government/whole-of-community approach, some DVs noted existing partnerships and connections that could be leveraged.
The DVs recognized misinformation, disinformation, and the distortion of information as increasingly prevalent threats and emphasized the need to fill resource gaps to combat these threats.
There was widespread agreement that the ACI and other cybersecurity and national security organizations should continue to hold exercises like JV and continue focusing on including state and local representatives in these valuable efforts.