JACK VOLTAIC RESEARCH METHODOLOGY

4. JACK VOLTAIC RESEARCH METHODOLOGY

4. JACK VOLTAIC RESEARCH METHODOLOGY

4.1. Introduction

This chapter describes the development of the JV 3.0 research and experiment design, both as a discrete event and as part of a research series. JV 3.0 began with initial concept development by the ACI in May 2018, continued through execution in September 2020, and beyond into 2021 with data analysis and the publication of this report. Because JV is primarily municipality-focused and includes as many private partners as possible, planning and execution for this research relies on coordinating a coalition of willing participants. The value of the event depends on engaged participation from individuals and organizations with diverse perspectives and scopes of responsibility. Maintaining broad engagement and ensuring all participants received as much value from JV 3.0 as they provided via their participation are some of the greatest challenges of performing JV research. As such, it is important to note that this chapter provides a synthesis of design concepts, evolving requirements, and key decisions which resulted in the execution events. Additionally, this section will mainly be a summary of final decisions and designs; more details are provided in the appendices for those desiring to see a more complete description of the process.

This chapter begins with a detailed explanation of each of the five principal research objectives, including a full description, the motivation behind the objective, and how it was incorporated into the overall JV 3.0 research event. These objectives and the objectives of the participants are the unifying thread for all aspects of the JV development and execution process. The remaining sections of this chapter detail the main components of JV, the partner organizations that played key roles in supporting the development process, the participating organizations whose input was critical to JV research, the scenario that the team designed to draw out the JV research objectives, the data collection plan, and the planning time line.

4.2. Research Objectives

Preparation for the JV events centers around the five principal research objectives. This section provides a more detailed description of each objective, insight into the motivation behind the objective, and how the ACI planned to incorporate the objective into JV 3.0. The objectives of JV 3.0 were to:

Examine the impact of a cyber event on Army force projection;
Exercise the cities of Charleston and Savannah in emergency cyber incident response to ensure the provision of public services and safeguard critical infrastructure;
Reinforce a whole-of-community approach in response to cyber incidents through sustained, multiechelon partnerships across industry, academia, and government;
Examine the coordination process for providing cyber protection capabilities in support of Defense Support to Civil Authorities (DSCA) requests; and
Support the development of a repeatable and adaptable framework that allows a city to exercise its response to a multisector cyber event.

4.2.1. Examine the Impact of a Cyber Event on Army Force Projection

JV 2.0 highlighted the challenges ports face in preparing for and responding to a physical or cyberspace attack. This project identified gaps in understanding of cyber threats, service-level agreements, and overall cyber response procedures. As a result, one of the primary research objectives of JV 3.0 was to examine the impact of a cyber event on Army force projection.

To study cyber incident effects on Army force projection, the ACI wanted to explore in-transit visibility considerations, including the identity, status, and location of DoD units, unit cargo, passengers, and personal property; understand Army movement control systems that regulate movement according to command priorities and synchronize the distribution flow of land forces; and consider strategic mobility activities that can mitigate the effects of natural and man-made obstacles that could hinder freedom of maneuver. The ACI also wished to assess whether the DoD is prepared to execute appropriate contingencies, branches, and sequels if Charleston and/or Savannah are unable to support force projection as a result of a cyber incident.

To observe the effects of a cyberattack on the fort-to-port supply chain in the JV 3.0 exercise, the ACI incorporated Emotet and phishing attacks into the exercise scenario. Emotet infected ships and trains’ cargo databases, and aggressive phishing attacks were directed at electricity and natural gas utilities. In addition, railway switching stations began to malfunction, resulting in further confusion and delays. To measure the effects of these disruptions on public and private sector operations and coordination, ACI data collectors observed the movement status of equipment; effects on military, civilian, and political decision making; effects on deployment; economic costs; vulnerabilities; response times; and secondand third-order effects on the mission (contingency planning).

4.2.2. Exercise the Cities of Charleston and Savannah

The focus of this research objective was to provide municipal organizations an opportunity to detect and respond to a cyber incident and assess their capability levels in that regard. Cities require the ability to provide adequate emergency cyber incident response to ensure the provision of public services and to safeguard commercial critical infrastructure. The ACI selected the cities of Charleston and Savannah due to their proximity to strategic ports that supported Defender 2020 and because both are in the same geographic region.

In the JV 3.0 experiment, the ACI wanted to identify and gauge cognitive, personal, and in-progress observations from the municipal emergency response perspective. In addition, the ACI wished to better enable municipal identification of potential gaps and threat vectors as well as response capabilities. The ultimate goal in this regard was to allow Savannah and Charleston to: (1) rehearse their capabilities and incident response plans; and (2) identify current gaps, critical infrastructure interdependencies, best practices, and existing resource allocations, thereby providing a common operating picture that informs the strengthening of cyber incident response mechanisms and bottom-up resiliency.

One of the intended effects of the scenario injects was to overcommit local public and private resources within the two port cities. Real-world examples were utilized to increase realism and believability. The responsibilities of the ACI data collectors were to capture key municipal perspectives on potential individual, resource, and framework gaps in responding to a debilitating regional event. ACI data collectors collected survey data from municipal critical infrastructure executives on their municipalities’ cyber incident response plans, capabilities, and resiliency; collected interview data to tell the story of JV from the municipality level or higher; and gauged and measured the response efforts of municipal emergency response organizations during the various turns in the scenario.

4.2.3. Reinforce a Whole-of-Community-Approach

A whole-of-community approach is critical for improving the detection of and response to a cyber incident; in a digitally connected environment, the owners of compromised systems and devices put not only their own information and infrastructure at risk, but also the information and infrastructure of other organizations that depend on their services. Vulnerability to cyber disruption is a whole-of-community problem that requires multi-echelon, cooperative action by governmental entities as well as private industry if it is to be solved. JV’s bottom-up approach focuses on a multi-echelon, cooperative approach to preparing communities that are highly likely to be targeted by malicious cyber actors.

The JV 3.0 scenario reinforced a whole-of-community approach to cyber incident response and critical infrastructure resiliency by allowing participants to observe the responses of other participants and to identify organizations and sectors with which they should communicate, thereby laying the groundwork for increased collaboration. The ACI wished to identify the municipal, state, federal, and industry partnerships, relationships, and collaborations that are necessary to support a whole-of-community approach to cyber incident response.

ACI data collectors focused on examining public-private partnerships; the internal capabilities, external support, and institutional knowledge of both government and industry; and the actions of participants from academia, including the involvement of institutional resources, federally funded research and development centers, nonprofit organizations, and think tanks.

4.2.4. Examine the Coordination Process for Providing Cyber Protection Capabilities in Support of DSCA

The focus of this research objective is to improve situational awareness of the DoD capabilities available to cities and states in the event they need support dealing with a cyber incident. DoD Directive 3025.18 establishes policy, assigns responsibilities, and provides guidance for the execution and oversight for DSCA, also referred to as civil support.20 DSCA support is provided by U.S. federal military forces and DoD civilians, contract personnel, and component assets. This DSCA directive also authorizes emergency authority for the use of military force—such as the National Guard—once Title 32 of the U.S. Code is invoked and requested through the lead federal agency. DSCA support can only occur once civilian capabilities have been exhausted and support has been requested by civil authorities. DSCA is evaluated under “C.A.R.R.L.L.,” which assesses the following aspects:

Cost: Who pays, and what is the impact on the DoD’s budget;
Appropriateness: Whether the requested mission is in the department’s interest to conduct;
Risk: The overall safety of DoD forces;
Readiness: Impact on the DoD’s ability to perform its primary mission;
Legality: Compliance with laws; and
Lethality: Potential need for lethal force by or against DoD forces.

²⁰ William J. Lynn III, Defense Support to Civil Authorities (DSCA), DoD Directive 3025.18 (Washington, DC: Office of the Secretary of Defense, December 29, 2010).

Regarding cyberattacks, cyber forces are not included in the execute order and not preprogrammed. Although governors and adjutant generals can provide cyber support to local governments and critical infrastructure using National Guardsmen in State Active Duty, all requests for cyber support under Title 10 go up to the level of the assistant secretary of defense as a result, after which a memorandum of understanding (MOU) or memorandum of agreement to send DoD support to the Cyber Protection Brigade is signed. All of this entails responding to physical impacts, assisting with halting the cyberattack and recovery efforts, and deploying forces (if required), all in coordination with other government agencies.

Though the DSCA process is well defined and understood at the local level, the process for requesting DoD support for cyber incidents through Defense Support to Cyber Incident Response (DSCIR) is not as clear. JV 3.0 sought to assist in clarifying and refining this process for all its participants through its Law and Policy TTX, an event that allows organizations to assess their cyber incident response plan, understand the composition and distribution of cyber assets across the state and federal government, and explore the authorities that govern the DSCIR process. Furthermore, the ACI tailored the scenario so that participants could exercise their incident command relationships, decision-making assistance, and information-sharing mechanisms.

4.2.5. Support the Development of a Repeatable and Adaptable Framework

This research objective, part of the JV research series strategy, is about making it as easy as possible for municipalities and private entities to conduct resiliency exercises while taking into account the conditions that make them unique. There are several processes publicly available for conducting exercise planning that include business, military, and even critical infrastructure. The ACI’s goal is to go beyond providing a general process by eventually offering a platform that will assist in the creation of distributable documents, rapid scenario development, and conducting of events. JV 3.0 was designed to serve not only as a research experiment, but also as the first template for future JV events, regardless of whether they are led by the ACI. Additionally, the ACI is working on automating much of the process to reduce the time investment for exercise planners and assessors.

Because the ACI is a military entity, relationship building was a key aspect of coordinating a coalition of willing participants. Participants had to trust each other and the ACI for JV to work because it required individuals and organizations to potentially expose some hard truths. More importantly, the ACI is constantly fielding requests for JV events throughout the country, and the team is simply too small and the time lines too long for the ACI to scale JV to meet the demand. The ACI cannot stress enough how grateful it is that all the participants engaged with it throughout the JV process, especially with their willingness to trust and be challenged; by making JV development accessible, the team hopes to enable emergency responders at all levels to leverage existing relationships for rapid and effective exercises. By keeping this research objective in mind throughout the process, the ACI hopes to design a tool that is easy to use, allowing cities of any size and budget to plan and execute their own JV events and ACI personnel to focus on conducting research and offering advice when necessary.

4.3. Components

JV comprises three components: the Planning Team, the TTX, and the live-fire exercise (LFX). The latter two are cyber simulations and dependent upon available resources (e.g., employee availability) and capabilities (e.g., access to IT and OT virtual range environments). Component 1, the Planning Team, is the most critical. Component 1 comprises representatives from sector-specific critical infrastructure organizations. The three-component structure gives participants the opportunity to conduct collective cybersecurity training, enhance cross-sector information sharing practices, coordinate technical-level threat information sharing, and communicate effects and risks to management. The LFX, which was canceled because of issues arising from the COVID-19 pandemic, would have exposed participants to threat tactics, tools, and shared techniques.

Component 1: The Planning Team comprises representatives from sector-specific critical infrastructure organizations. Team members, also known as “trusted agents,” are key to successful development and execution.
Component 2: The TTX is a simulated and facilitated discussion based on a scenario that takes participants through the process of dealing with a physical disaster blended with malicious cyber activities. Note: This component will be described in greater detail in section 4.7, “Scenario,” and Appendix D, “Law/Policy Tabletop Exercise (TTX).”
Component 3: The LFX is a JV exercise component that uses an on-range, simulated, virtual environment. The LFX follows a scenario that correlates with the TTX scenario. It exposes participants to threat tactics, tools, and shared techniques and tests cyber equipment and response capabilities in real time. Note: Due to circumstances beyond the control of anyone planning the event, the LFX was not included in this iteration of JV. It is included in this report because it remains a critical component for the future.

The Planning Committee is the most critical component to the successful planning and execution of a JV event. The JV 3.0 Planning Committee included governmental representatives from Charleston and Savannah, sector-specific critical infrastructure organizations, and the ACI and its partners. The members of the committee took part in one or more working groups, or Operational Planning Teams (OPTs), that directly supported key aspects of JV 3.0. The following are the names and purposes of the OPTs:

Lead and Resource Support: Plan, resource, and coordinate the OPTs to support both the OPTs’ purposes and the event’s overall objectives.
Scenario Design and Execution: Design and execute an objective-focused event with a realistic and integrated scenario with injects focused on participant-nominated objectives.
Data Collection and Analysis: Identify, understand, collect, assess, and synthesize impactful qualitative and quantitative data that both supports bottom-up resiliency and ensures Army force projection capabilities.
Cyber Range Development: Provide a combination virtual/physical space in which JV event participants can conduct a cyber game scenario using realistic representations of municipality infrastructure.
Law and Policy TTX: Baseline understanding and address underlying concerns about authorities, reporting, and assistance.
Strategic Communications: Effectively communicate the meaningful stories and messages of JV to key audiences. 
Distinguished Visitor (DV) Day: Create an opportunity for Senior Leaders and Executives to experience JV. Note: Due to the COVID-19 pandemic, this OPT shifted to an Executive Out-Brief to provide senior leaders and executives initial feedback on JV 3.0.

Planning for JV 3.0 included multiple planning and rehearsal workshops. The early workshops were designed to start with relationship building and solicit objectives from prospective partners and participants. Later workshops solidified the scenario, confirmed participant rosters, validated that participant objectives and JV research objectives were being addressed, and gave participants multiple opportunities to practice with the technologies that were going to be used to administer the JV events.

4.4. Planning Time Line

The ACI and its partners held a series of planning meetings and workshops that facilitated establishing the membership of the Planning Committee, understanding stakeholder objectives for the exercise, and developing a scenario that would meet stakeholder and event objectives (see Figure 2, “JV 3.0 Planning Time Line”). Prior to March 2020, the Planning Committee met in person for most meetings and workshops. These on-site events allowed the members of the committee to develop strong relationships and trust that eased the transition to virtual events after pandemic-related restrictions took hold.

When the Planning Committee shifted to a virtual execution, they recognized two key challenges: maintaining stakeholder engagement and increasing participant comfort with the required technology. The ACI, NUARI, and FTI Consulting sought to address these challenges by providing stakeholders and participants an opportunity to participate in three separate virtual TTXs. The first, Jack Pandemus, was a 3-hour event that served as a test for virtual execution using both NUARI’s DECIDE® and Microsoft Teams. Following Jack Pandemus, the ACI and its partners held two additional 4-hour events using DECIDE® and Microsoft Teams. These rehearsal events allowed the Planning Committee to refine its execution plan and provided participants additional opportunities to gain experience with the event and the various supporting platforms.

Figure 2: JV 3.0 Planning Time Line

4.5. Partners

The ACI works with partners that have mutual interests and that aim to resolve similar issues. Preventing future cyber-related crises can become a reality through establishing public-private, academic, and industry relationships with relevant experts. Furthermore, JV 3.0 and Jack Pandemus (see section 5.1) would not have been possible without these partners.

All partner contributions were truly invaluable and necessary for conducting JV 3.0. Without these partners, JV 3.0 would not have been a success. Full details of each partner’s contributions are included in Appendix B, “Partners.” The following is a summary of each organization’s contributions to JV.

4.5.1. City of Charleston

The City of Charleston, SC, participated in both the planning and execution of JV 3.0. Beginning with the JV 2.5 workshop in May 2019, city staff learned about the history and goals of the JV program. Staff also gained a better understanding of why the ACI and its partners were interested in studying the current security posture of municipal governments and an appreciation of their working relationships with state and federal agencies with respect to the identification of, management of, and response to cyber events. Charleston assisted in identifying potential participants in the region from both the public and private sectors who would likely be impacted by a significant event in the area. In the planning meetings, city staff provided information about the unique geography of the region and interactions among various local agencies, with the goal of providing background information to allow for the creation of an exercise scenario that was germane to the local participants. Representatives from the city’s Information Technology, Traffic and Transportation, Police, and Fire departments, among others, actively participated in the TTX.

4.5.2. Town of Mount Pleasant

The Town of Mount Pleasant, which is located across the Cooper River from Charleston, provided integral support from the JV 2.5 Cyber Workshop Series through the planning and implementation of JV 3.0. In addition, Mount Pleasant provided leadership in decision support for the unique challenges arising from the impacts of COVID-19. Mount Pleasant’s emergency manager served as a regional point of contact for the exercise and ensured participation from stakeholders and partners. Early on, Mount Pleasant hosted events with critical infrastructure representatives to introduce the ACI and collaborate on SC cybersecurity concepts and issues. These events allowed for the sharing of unique regional insights and provided groundwork for the initial planning phases of JV 3.0. Furthermore, Mount Pleasant’s Information Technology department worked in concert with the City of Charleston to coordinate information sharing among players and stakeholders. This work strengthened Mount Pleasant and Charleston’s partnership and overall cyber readiness and posture.

4.5.3. City of Savannah

The City of Savannah, GA, was involved early in the planning process. Led by the City of Savannah emergency management director, the IT, emergency preparedness, fire, and water resources departments became significantly involved in the planning. Savannah’s emergency manager and IT department served as the city’s points of contact for the exercise, introducing ACI to critical stakeholders in the area. In addition to supporting and attending the ACI meetings, Savannah held its own internal meetings to discuss and determine participation. The city also finalized its Cyber Incident Annex as part of its preparation. Savannah had 18 personnel from multiple agencies participate in the Rehearsal of Concept (ROC) Drills and exercise. The local police department participated in the ROC Drills, but it could not make the final exercise because its participation was preempted by a real-world incident.

4.5.4. FTI Consulting

Because JV research emphasizes the need for public-private partnership and a whole-of-community approach, the ACI recognizes the need for private expertise to make JV events valuable. To that end, the ACI was proud to partner with FTI Consulting for JV 3.0. FTI Consulting is a global business advisory firm dedicated to helping organizations manage change; mitigate risk; and resolve financial, legal, operational, political and regulatory, and reputational and transactional disputes. More than a dozen FTI Consulting team members, including senior executives and sector subject matter experts, participated in JV 3.0. As recognized leaders in organizational cybersecurity from both technical and policy perspectives, FTI Consulting provided invaluable planning, leadership, and technical expertise to the development, execution, and publicity of JV 3.0 throughout the process, especially in the areas of scenario development and executive communication.

4.5.5. NUARI/DECIDE®

NUARI partnered with the ACI for JV 3.0. NUARI is a 501(c)(3) nonprofit organization that serves the national public interest through the interdisciplinary study of critical national security issues. NUARI is partially funded by DHS and the DoD and federally chartered under the sponsorship of Senator Patrick Leahy. NUARI provides cyber exercises, secure network monitoring, custom consulting, research, and education through many avenues, including its DECIDE® platform exercises. During planning, the DECIDE® platform was intended to serve as the primary means of distributing full information about scenario events and capturing participant responses, with in-person facilitation and conversation serving as an alternate means. When COVID-19 effectively prevented all in-person events, the DECIDE® platform became the sole platform for conducting the virtual TTX. JV 3.0 would not have happened without DECIDE®.

4.5.6. AT&T/FirstNet

FirstNet is the Nationwide communications platform dedicated to America’s first responders and public safety community, built with AT&T in a public-private partnership with the First Responder Network Authority. Prior to the pandemic, AT&T worked with the ACI to provide a full suite of advanced tools that would serve as the contingency communications infrastructure for the JV 3.0 exercise. These tools included two satellite on light trucks (SATCOLTs), 60 FirstNet-enabled devices, projection monitors, and a team to support the ongoing communications among the participants from local, state, and federal entities. AT&T also planned to provide the ACI with a video team to capture each exercise incident as it unfolded to create a documentary of the events in Charleston and Savannah. When the JV 3.0 event was moved to a virtual format due to COVID-19, AT&T provided a team of subject matter experts in emergency communications, who participated in both the online event and numerous planning sessions to educate the participants and provide guidance on crisis communications, restorative procedures, and FirstNet. AT&T’s participation in the planning, execution, and data analysis contributed greatly to the quality of the event and this report.

Figure 3: This AT&T SATCOLT is one of the tools that would have served as contingency communications infrastructure for JV 3.0 pre-COVID-19

Figure 4: This AT&T cell on wings, also called a “flying COW,” would have been one of the FirstNet-enabled devices provided during the JV 3.0 LFX.

4.5.7. Intrepid Networks

Intrepid Networks provides Intrepid Response, a FirstNet-certified and affordable web and mobile situational awareness software platform for day-to-day and emergency operations. Originally, the ACI partnered with Intrepid Networks to furnish licenses to use Intrepid Response on FirstNet phones that would have been provided by AT&T to participants. This would have provided an additional common operating picture platform to achieve realism during the TTX. Intrepid Networks continued to partner with the ACI after the in-person events were canceled and generated exercise common operating picture maps that coincided with scenario events, giving participants the ability to engage with the scenario based on specific urban geography; this achieved an effect like that of Intrepid Response. Intrepid Networks’ contribution significantly improved the quality of engagement and the realism of the scenario.

4.5.8. The Citadel

The Citadel hosted a JV 2.5 workshop in Charleston on May 21, 2019. The college worked with the ACI to organize the workshop. In addition, faculty from The Citadel supported the planning efforts, attending the JV 3.0 Initial Planning Meeting in Augusta, GA, on July 9–10, 2019; numerous planning workshops; and the ROC Drill for Charleston on September 8, 2020. Faculty and students from The Citadel participated in the exercise itself, serving as both participants and data collectors.

4.5.9. Savannah Technical College

The ACI and Savannah Technical College (STC) began working together in January 2020. STC provided academic advisory support and facilitated face-to-face meetings prior to COVID-19. Also prior to COVID-19, the ACI and key partners completed a site visit and chose STC as the on-site location for the Savannah JV 3.0 exercise. More than 15 students registered to help as data collectors for the Savannah iteration. This both facilitated the success of JV 3.0 data collection and allowed students to gain valuable knowledge and insight into an aspect of cyber readiness needs and methods that could not be taught solely in the classroom. In addition, STC served as a member of the Distinguished Visitor Day and Scenario Design and Execution OPTs. In the future, STC will continue to collaborate with the ACI by incorporating the JV experience into future training exercises in the coastal GA region.

4.6. Participants

Table 1: JV 3.0 Participants

4.7. Scenario

Information overload is a serious problem with which to contend in both real-life emergency response and fictional exercises. Policies and procedures regarding information sharing are often crafted to streamline distribution of preidentified information types to the most relevant parties. Experienced personnel therefore know and handle much more than is communicated. Because of this filtering of communication, prebuilt relationships are extremely valuable. However, when truly new situations arise for which there are no established policies or practical experience available, information sharing can be slow and inappropriately distributed. Highlighting this difficulty, previous JV events and workshops have revealed cyber incident policies and information sharing agreements that are often incomplete or nonexistent. This was the impetus behind the creation of the JV scenario.

The scenario was the primary method for pursuing the research objectives. Because JV brings together individuals and organizations with diverse and valuable expertise, and no one organization is the single source of expertise and best practices, bringing everyone together to play a fictional game is often the best way to tease out relevant knowledge from the best people in place to handle emergency situations. These conditions create a collaborative learning environment in which we can pursue our research objectives.

4.7.1. Design Requirements

The scenario design needed to accomplish many goals simultaneously:

Support both the event research objectives and the participant objectives.
Maintain realism. All injects included in the scenario, especially cyber incidents, were either sourced from real events or forecasted in scholarly works. This ensured relevance and minimized the threat of participants balking at the scenario and refusing to participate.
Achieve ambiguity regarding severity and the cause of the damage, whether it was equipment failure resulting from normal physical degradation or a cyber intrusion. In other words, the cyber incidents in the scenario needed to avoid being obviously cyber-related.
Achieve ambiguity regarding the level of sophistication of an actor. In other words, the cyber incidents needed to not be so sophisticated that only a nation-state actor would be capable of performing the attack.
Keep incidents below a threshold of armed conflict.
Focus cyber intrusions on local municipality and private entities.

Additionally, the designed scenario introduced a certain level of stress prior to the cyber incidents. Because an adversary would most likely time its intrusions and disruptions to have a maximum impact, it was important for local resources to already be in place to deal with other, noncyber issues. For this reason, protests, traffic issues, and natural weather considerations were included in the scenario to ensure participating emergency responders were already expending planning, personnel, and materiel resources before the additional events occurred.

4.7.2. Design Concepts

In designing the scenario, the Planning Team’s strategy was to use injects that progressively built upon one another, avoid introducing attribution, and keep incident causes ambiguous for as long as possible. This “death by a thousand cuts” approach allowed the ACI and its partners the opportunity to explore thresholds at which organizations would identify a cyber incident and request support. Keeping the cause of the incident ambiguous facilitated debate among participants, encouraged them to share their decision-making processes with other participants, and increased the realism of the exercise.

The scenario was designed to be played over a series of turns and to weave together multiple independent threads—a set of sector-specific injects that build on themselves—to form a cohesive story. Each thread was built such that its specific injects would grow progressively more dangerous, either by spreading to new areas, organizations, or systems or by causing increased amounts of damage to affected entities. During the planning workshops leading up to JV 3.0, it was evident that many participating organizations, particularly in the municipalities, lacked the resources to adequately defend against a sophisticated adversary. Therefore, the Planning Team designed the scenario from a perspective of assumed compromise. Many of the scenario parameters, such as when malware exploitation would migrate from sector to sector, were deliberately kept opaque to the players. This approach forced participants to respond to incidents rather than attempt to defend against them. See figure 5 for a graphical display of the expected progression.

Figure 5: JV 3.0 Scenario Development Framework

Following this design philosophy allowed several important benefits:

Creative freedom could be given to multiple independent scenario writers, each with his or her own expertise (for example, the energy sector), without hindering other writers’ efforts.
Starting small with each thread ensured no one thread would dominate the scenario because of how it was written. Thus, if the scenario incidents caused the conversation about one specific thread to become dominant during the JV 3.0 event, this would be useful information for data collectors.
There was more going on in the scenario than participants could see. Because DECIDE® was able to distribute injects to participants based on their roles and responsibilities, any thread that was not discussed due to lack of participation would simply not be part of the conversation. This ensured that all players were able to participate based on their personal and expected expertise, without relying on players to inexpertly speculate on the activities of organizations not able to participate.
The slow progression of each thread meant the overall scenario difficulty would increase incrementally from turn to turn, thereby allowing a more organic discussion of the thresholds for responses, declarations, and requests.
This algorithmic approach makes it possible to combine any number of independent threads to create unique scenarios quickly, depending on focus and need. This benefit supports the automation project discussed later in this report.

4.7.3. Validation

During the leading workshops that occurred throughout the planning and development process, the Planning Team tested the scenario. Through repeat rehearsal and refinement, the Planning Team not only validated each individual thread, but also provided the basis for understanding expected responses to the scenario elements. This allowed the scenario development team to build a realistic and challenging scenario that ultimately maintained engagement during the JV events and supported the data collection and analysis to successfully address the research and participant objectives.

4.8. Data Collection and Analysis Plan

JV 3.0 incorporated a stakeholder-driven, multipronged data collection approach. The primary goal was to collect and analyze meaningful data to help build critical infrastructure and emergency response capacity and resiliency at a municipal level and to inform Army tactical, operational, and strategic calculations regarding potential impacts on force projection capabilities. Accordingly, the data collection and analysis plan was designed to identify critical information that could help answer the overarching JV 3.0 research objectives previously identified. The following sections go into greater detail about these objectives.

4.8.1. Developing the Data Collection and Analysis Plan

Creating an effective data collection and analysis plan required the identification of key stakeholders and information requirements for each research objective referenced above. Each of the three JV 3.0 planning workshops included sessions for developing and refining data collection and analysis procedures to ensure stakeholder critical information requirements were identified during the preliminary planning and design phases of the event. Once the relevant issues were identified for key city, county, state, federal, military, and private sector stakeholders, an information synchronization matrix was constructed to visualize appropriate indicators and information requirements to support the achievement of the research objectives. Additionally, key supporting stakeholder requirements were outlined to facilitate better understanding of all potential areas for data collection and analysis efforts during JV 3.0. This stakeholder-informed methodology resulted in the identification of: (1) specific information requirements to support primary JV 3.0 research objectives; (2) a coalition of willing stakeholders to help support data collection and analysis; and (3) potential gaps in data collection and analysis.

The critical steps in plan development included:

Identifying strategic research objectives;
Nesting stakeholder objectives within this strategic research framework;
Determining data objectives and synchronizing intersecting areas of interest;
Cataloging all available resources in support of data collection and analysis to identify redundancies, interdependencies, and potential gaps;
Verifying essential elements of information, key indicators, and methods of collection on available platforms; and
Designing the most advantageous data categorization scheme to facilitate post-event analysis and support the generation of the final report.

4.8.2. Workshops and Stakeholders

Creating a coalition of willing partner organizations was a key facet of the JV 3.0 data collection and analysis. These partner organizations were integral contributors to survey question design, organizational data collector support, data postprocessing, data visualization, and the production of key areas of this final report. To further synchronize and enhance this support given the change to a distributed execution, the ACI hosted three data collection and analysis workshops prior to the JV 3.0 exercises with participation from state, military, academic, and private sector partners. The key takeaways of these workshops are detailed below.

Workshop #1: June 2020

Reaffirmed and further solidified data collection and analysis partnerships for JV 3.0 as the team worked toward event execution;
Generated additional participation and support for data collection and analysis during the Jack Pandemus mini-exercise (described later in this report);
Created new partnerships both for the ACI and within the larger data collection and analysis team;
Ensured a clear understanding of and consensus on data collection and analysis efforts before event execution; and
Established redundancy in collection platforms, methods, and constructs to ensure a robust dataset for holistic post-event analysis.

Workshop #2: July 2020

Reaffirmed and further solidified data collection and analysis partnerships for JV 3.0 as the team worked toward event execution;
Identified lessons learned and areas for refinement following the execution of the Jack Pandemus exercise; and
Created a common operating picture of holistic and robust support for JV 3.0 data collection and analysis.

Workshop #3: August 2020

Revalidated commitments and updates for partners and participants;
Finalized data collection approaches, platforms, and tools to be used during the JV 3.0 event;
Presented and discussed survey question development, methodology, refinements, comments, and recommendations prior to event execution;
Refined and recommended final data collector guidance; and
Established an additional working group to support partnerships and new ways to facilitate additional collaboration going forward.

Numerous stakeholder organizations participated in these workshops and volunteered to support data collection and analysis planning, execution, and postevent efforts. Participating organizations included:

U.S. Army War College
University of Illinois at Urbana-Champaign (UIUC) Critical Infrastructure Resilience Institute (CIRI)
NUARI
Idaho National Laboratory (INL)
Johns Hopkins University Applied Physics Laboratory
3rd Infantry Division (3ID) Headquarters
Military Surface Deployment and Distribution Command (SDDC)
597th Transportation Brigade
841st Transportation Battalion
Center for Army Analysis
Intrepid Networks
SC Law Enforcement Division (SLED)
FTI Consulting
The George Washington University
Provatek
Blank Slate Solution

Validation and continuous refinement of the data collection and analysis plan occurred across multiple smaller events leading up to the JV 3.0 events. Validation and proof of concept events included:

Jack Pandemus—Pandemic-based cyber incident scenario exercise distributed through the DECIDE® platform and Microsoft Teams.
- » Allowed for initial validation of survey question structure and delivery and DECIDE® platform data collection functionalities.
ROC Drill #1—Initial scenario delivered to participants in a controlled environment in preparation for event execution.
- » Revalidated survey question structure and delivery and DECIDE® platform and Microsoft Teams meeting process and data collection functionalities.
ROC Drill #2—Initial scenario delivered to participants in a controlled environment in preparation for event execution.
- » Finalized survey question structure and delivery and DECIDE® platform and Microsoft Teams meeting process and data collection functionalities.

4.8.4. Data Sources

To facilitate robust data collection, multiple platforms and functionalities were built into the overarching data collection approach, as described in table 2.

Table 2: JV 3.0 Data Sources

4.8.5. Data Coding Schema

A data classification coding scheme was developed for categorizing the exercise observations to assist in postprocessing efforts following execution. Data type tags, classification descriptions, and examples were outlined for data collectors prior to the event and included in the Jack Voltaic 3.0 Data Collector Guide distributed to all volunteers. Some of the data tags used in the JV 3.0 exercises are listed in table 3.

Table 3: JV 3.0 Data Tags

4.8.6. Data Collector Guidance Development and Training

Data collection and analysis planning culminated in the Jack Voltaic 3.0 Data Collector Guide, which was distributed to volunteer data collectors. The guide includes primary data collector responsibilities; classification codes for DECIDE®; and a common concept for capturing data, platforms, and mechanisms.

Additionally, the Data Collection OPT conducted training sessions for volunteer data collectors and exercise controllers to ensure proficiency in platform features and data coding and familiarity with virtual table assignments. The second training session, which included an overview of the hypothetical scenario, allowed data collectors to practice logging data into DECIDE® (row 4 in Table 2, “JV 3.0 Data Sources”) using the data codes as they listened to a mock discussion among three “participants” (members of the JV 3.0 organization team).