Introduction to applied research methods 

 by simulating a development project  (RD1)

Bo Strangert

This is a short summary of a case approach to construct a challenging learning task for students of applied research methods. It is about the possibility of improving the capability to plan, manage, and evaluate complex development projects by using an integrated Action Research & Development strategy.

The background of the case is a recurrent course in applied research methods commissioned by the Nordic Defence Cooperation (NORDEFCO). The course has been one element in a progressive international movement to develop military capabilities through improved experimentation. Albert and Hayes (2002) formulated a framework in their book Code of best practice for experimentation, and coined the term ‘Experimentation campaign’ for a series of related activities of conceptual development (CD) and experimentation (E) over time. A Nordic version of this framework has been published later as a NORDEFCO (2012) initiative.

This framework contains nothing new about scientific methodology, though its strong emphasis of scientific support to complex development work will probably have an impact on investigating corresponding issues even outside military practice.

Our immediate focus in 2011 was on how we could improve teaching a group of development officers to make them ready to apply basic methodological principles in their own live development projects within a short period of allotted time. During three former courses, the diversity of the participants’ projects had led to problems of bringing forward the essential interaction between the methodological and contextual aspects. Some comments in Swedish about such problems are found in Strangert (2013).

A possible solution was to construct exercises about a common fictitious practice case as a guideline for their own project applications. But how should an ideal case be constructed that encompasses all necessary methodological and other challenges about complexity? A simulation of a natural development process should allow participants to make creative initiatives, when confronted with a wealth of uncertain opportunities.

Obviously then, an COBP (Code of best practice) for application of abstract principles to complex cases would be too general as a single guideline for exercises. On the other hand, a work method for requirement specification of military products could be too ordered and specific. That’s because we wanted to prioritize validity issues at the expense of verification requirements.

Case study of a simulated development project

I outlined a general structure for the development process, including a real life context, a broad goal description, and a few preconditions about the development team’s forms of working and exercises. The resulting actual direction and content of the process would partly depend on the participants’ competencies and achievements. Thus, the purpose was to direct their learning towards essential capabilities through recurrent exercises of identifying and formulating problems, solution attempts, and feedback.

Real-life context and development goal. The chosen context was the political, strategic, and tactical challenges today for peacekeeping missions in foreign countries, including both military operations and civil-military cooperation for development aid.

The overriding development goal was the actual dilemma of combining military effectiveness with the aspiration to reduce collateral damage to an absolute minimum. The devastating consequences of collateral damage have become an increasingly hot issue – ethically, humanely, and politically. From a military point of view, the means of handling the problem should include decisions and actions on all levels: strategic, operational, tactical, and technical.

Preconditions. The participants constituted a development team, with roles changing according to tasks, competencies and preferences. The initial development task was to mind the well-known problem of effectiveness and collateral damage on all levels, from strategic to technical, and to review different sources of knowledge. Therefore, the participants got a chart for common planning.

The planning model. I constructed a multidimensional levels-of-analysis model for planning purposes. Essentially, it is an extended partitioning of the basic scientific distinction between theoretical and empirical levels of representation. Its core is a project space of planned or executed CD and E operations along a time axis.

A projection of the model onto a plane (Figure 1) shows that the model outside the project space also includes a level of associated structured knowledge domains and an empirical reality space. Two additional levels of dynamic cognitive and communicative dimensions are not illustrated in Figure 1.

Figure 1. Planning structure represented by the level-of-analysis model.

The planning process. Its general purpose was to stimulate divergent and convergent thinking and step by step attain more mature project states regarding the preconditions and the project goal. Exercises were designed to promote advancement according to a subgoal structure. However, this subgoal structure had itself to be constructed gradually, through control cycles of test-actions-feedback with double loops. Thus, the double loops included tests of actions  as well as associated subgoals. Hence, the design logic has some similarity with program evaluation. The major difference is the concomitant active construction and reconstruction of the program itself.

The prime outcome of the planning process is the interplay between CD and E through rational-analytical processes and empirical testing of hypotheses and conjectures. Figure 2 illustrates schematically a set of stages of CD and E activities.

Figure 2. Schematic illustration of CD and E activities during a project . 

Planning stages

Structuring the initial planning process

To maintain an open mind to solutions, the idea of possible reduction of collateral damage without hampering military effectiveness deserved careful preliminary consideration. It involved a general conceptual goal analysis, based on documented knowledge and expertise on possible measures. It also called for collection of new data and analyses about former and present events and experiences on risks for collateral damage.

The goal analysis suggested a preventive approach, by improving the enabling capabilities of reconnaissance, surveillance, and target acquisition (RSTA). Possible effects of new procedures should be that the risk of collateral damage was reduced without serious consequences for other political and military purposes.

Furthermore, the target area for the RSTA procedures should be operations in the urban terrain, including dense civil populations and fragile infrastructures.

These CD and E activities required foci on different aspects of the task. The levels of strategy, operations, tactics, and technique were one important dimension. Another concerned the connections between these levels. A preliminary division of the CD space led to three part-projects: one (B) with a micro perspective on man-machine solutions, another (C) with a macro perspective on interoperability and strategic questions, and a central part-project (A) on low-level command and control in army and air forces operating in the urban terrain and space. 

Coordination of design issues about technical, methodical, and personnel matters became a distinctive feature from the beginning of the project.  Technical design has apparent advantages in comparison with, for example, development of tactical or personal inventions, and usually takes the lead. This is not appropriate in complex systems, because methodical and personal issues, e.g. about C2, present the most difficult and decisive design tasks. Therefore, the part-project A took the lead in our case.

Initial experimentation and selection of modelling techniques

A new project should build on successes and failures of past experiences. It was decided to collect new data and reanalyze old knowledge about crucial actors and events in risky urban operations (UO). Particularly, advancing or patrolling infantry and air forces supporting such advances were suitable subjects for experiential data. 

The experimentation had to be based on appropriate conceptual structures. In part-project A, for example, an infantry commander’s C2 task was represented by a simple dynamic control model. It included, among other things, how the decision making was influenced by information from own forces, close air support (CAS), and higher-level command.

The methodological exercises covered design of surveys and in depth interviews of commanders with experiences of UO. Documents about incidents were to be collected and analyzed regarding risk and effectiveness measures.

Compared with A, the conceptual modelling in part-project B could be much more detailed and follow a technical work method for requirement specifications. The design task built on the idea of a small unmanned aerial system (UAS) for close RSTA by ground forces on squad, platoon, or company levels – a fictitious complement to air planes and drones. (Later, I became aware that similar devices were already projected in military industries, and a simple toy version was for sale in Apple store!)

The CD-exercise included a basic structural description of the UAS functions, that is, how its external sensor functions, through display and manual control functions, were coupled to the human operator functions. The exercises required the participants to specify necessary physical and operator attributes and parameters to be researched and designed. Exercises on experimentation focussed on laboratory experiments and physical measurement for requirement specifications.

Part-project C on interoperability involved a network representation of C2 including both collateral forces (e.g. air surveillance and strike capabilities) and higher staffs. The E-exercises focussed on planning of group sessions with high-ranking officers and domain experts, who had the task of evaluating project results and making suggestions and decisions about future advances. In particular, the exercises focussed on use of meetings for knowledge acquisition versus for interventions in the development process.

Analytical and experimental coordination of part-projects

The period of initial examination of problems and possibilities within part-projects turned to analyses of how to coordinate them. An integrated complex system for improved RSTA in urban operations requires well-defined interconnections between its subsystems A, B, and C. They must be designed so they will correspond to the ultimate goal.

This means recurrent pairwise and triple adjustments of the subsystems functions and parameters. For example, the commander’s C2 task requires adjustment of UAS’ physical and operator characteristics, and vice versa. The network functionality of interoperability may pose some restrictions on the commander’s use of UAS for C2, and so on.

Experimentation should be iterative and contingent on evolving results. The associated exercises involved design of suitable laboratory experiments and experimental simulations with time-series designs to identify critical junctures of integration. The systematic manipulations included, for example, target attributes for detection and identification of combatants and noncombatants, the commander’s work load, forms of communication, and many more factors.

The introduction of context uncertainty and dynamics in CD and E.

CD&E had until now followed the most clearcut and determinate conceptual implications of the ultimate goal concept. It entailed dealing with RSTA-improvement as a means of reducing collateral damage while maintaining military effectiveness.

Thus, a few preliminary steps had been taken to answer the central question of construct validity: how the system should be designed so that its behavior and effects would correspond to the stated ultimate goals in real-life context. Also, some basic causal preconditions had been tested in isolation.

In the next development step, we introduced contextual variation. It aimed at discovering circumstances that could be critical for the ultimate construct validity and generalization of the project results.

Scenario construction was used as a pseudo-empirical way of representing conceptual structures. Scenarios have been defined as stories that describe different but equally plausible futures; they are developed by using methods that systematically gather perceptions about certainties and uncertainties.The exercises included both normative (mapping present structures) and exploratory scenes (about future worlds).

Conceptual goal formulation guided the constructions. For example, consider a simple hypothesis that use of UAS for RSTA in combat will give effects on collateral damage and military effectiveness, depending on the task attributes of urban operations. The contextual variation of UO includes task attributes. To design suitable scenarios we used intelligence preparation of the battlefield (IPB). The first step was to define the battlefield environments to identify our intelligence gaps in order to prepare our RSTA plan. Here it was necessary to remember that the environment was uncertain as well as multidimensional, dynamic, and nontransparent.

The second step involved effects of different attributes as urban terrain, infrastructure, noncombatants, and weather. In the third step, the threats of the enemy were evaluated: characteristics, equipment, tactics. Finally, possible threat courses of action were determined against the background of step 1-3.

Experimentation aiming at ”vicarious validation”

Scenarios prepared by IPB were used in planning various simulations and quasi-experiments about possible causal attributes and courses of actions. The corresponding planning exercises included judgments about the prospect of obtaining acceptable construct, external, and internal validity in a real context. Such types of judgment of results from analytical calculation or empirical tests are of course essential. They form the basis for continued design of solutions to attain the ultimate goals.

Advanced case study design for evaluation

The course ended with focus on post-evaluation design and advanced case studies of systems in their naturalistic contexts.This means applying a retrospective view by explaining present behaviour in terms of its development history. It should make us more keen to apply scientific experimentation in future development approaches.

The last package of exercises was about how to conduct an advanced case study of the new RSTA-system in operation to evaluate its validity in real-life context.

Background. The synopsis concerned a UN mission in a foreign country with one army brigade involved in urban operations against armed insurgents. One UN front company was trained and equipped with new UAS-devices.

A recent course of events required an urgent investigation. The front company had clashed during an advance with enemy forces and the incident eventually resulted in a catastrophe with many civilian casualties and destroyed facilities.

The responsible UN head of the mission announced that: ”An independent investigation shall be conducted promptly to secure evidence about the course of events. It is of vital importance that the investigation is carried out according to scientific principles of objectivity and validity. The results should be trusted by all parties involved.”

An R&D team was composed to meet the need of different expertise.

The R&D team.  The team included nine development officers with the following specialist domains (special focus within parenthesis): Strategic, operational, tactical coordination (transfer of info between civil and military parties and across C2 levels), Interoperability (coordination of ground and air forces), Intelligence (Blue and Red forces’ respective access to intelligence), Tactical operations (actions of the involved units), RSTA-functions (methods of active monitoring the course of events), Logistics (command & organization of medical service and necessities), NBC (detection and identification of NBC contamination), Mission evaluation (documentation and evaluation design).

Plan of investigation

The exercise required the R&D team to make a joint plan to

– Formulate explicit research problems

– Develop a formal research design

– Use theory and empirical facts to develop specific  questions and hypotheses.

– Select methods for data collection

– Assemble a database that can be assessed by independent parties

– Conduct qualitative and quantitative analyses of data

Comments on the plan and its implementation

Due to the complexity of the evaluation task, the implementation was partitioned into several part-studies, before assembling and integrating the results and making final conclusions. Step by step the R&D team got fictitious case data to complete the exercises.

Obviously, investigations of complex phenomena seldom result in simple, unambiguous conclusions. However, the data form complex patterns of indices, which can be used for testing rival hypotheses and for directing further development attempts.

The plan followed closely the approach to case study research by Yin (2002), based on the fundamental principles of securing reliable and valid results. As Campbell succinctly puts it in the foreword to Yin’s book: ” It epitomizes a research method for attempting valid inferences from events outside the laboratory while at the same time retaining the goals of knowledge shared with laboratory science.”

A final note

This complete set of case exercises followed the development steps from an original idea and its conceptualisation, and treated ways of using experience and domain knowledge to form conceptual models. The continued CD & E eventually shaped and tested the complex system to be implemented and validated. Nevertheless, a complete goal attainment is never possible, because so much complexity, uncertainty, and dynamics are involved. Thus, validation and quality improvement must be continued during the system’s whole life-cycle. Its history contains important traces of its development.

In coming papers on this site, I will treat some of the methodological problems in this military case application more thoroughly. I welcome your comments on this and future papers!

References

Alberts, D.S. & Hayes, R.E. (2002). Codes of best practice for experimentation.   CCRP publication series.

Nordic Defence Cooperation (2012). CD&E Method Description. Version 2.

Scriven. M. (1964). Views of human nature. In Wann, T.W. (Ed.) Behaviorism and Phenomenology. Chicago: The University of Chicago Press, 1964.

Strangert, B. (2013). Om studieformer för att utveckla förmågan att tillämpa vetenskaplig metodik i professionell verksamhet. www.arborg.se/tillampning/T4.html

Taleb, N.N. (2010). The Black Swan. The impact of the highly improbable. London: Penguin book.