Appendix 1: Methods and Material and Reflections on the Research Process

From Initial Assumptions to Key Conclusions

This volume is one of the outcomes of the authors' joint project “Towards an organizational theory on obsessive measurement disorder,” funded by the Swedish Research Council. As mentioned in the introductory chapter, an important point of departure for the project has been to explain when, how and why obsessive measurement disorder (OMD) occurs or not, and to identify coping mechanisms and responses that may help to prevent overregulation and control, and instead foster constructive learning that benefits aid results.

Drawing on rich in-depth case studies of aid bureaucrats in inter-organizational relations, we set out to learn more about how and why decision-makers respond as they do to the high levels of complexity and uncertainty that characterize their operations. Needless to say, we have also been curious about the outcomes and consequences of these aid efforts, and especially keen to understand if, when, and how attempts to demonstrate certainty become counterproductive for the purposes at stake.

As our research journey progressed, however, we found that the core question of OMD could not be explained without taking on the broader topic on how aid bureaucrats deal with uncertainty. Throughout the research process, we have therefore broadened our scope and, as we learned more, adjusted our methodology and analysis accordingly. Below, we provide an account of the choices and the iterative process that led us to the findings presented in this book.

As a general note to the reader, we welcome that case findings presented and discussed in this volume are confirmed, or perhaps disproven, in future studies, including studies on larger populations of organizations and aid bureaucrats. At the time of writing, we have begun such a follow-up study, in a second project financed by the Swedish Research Council: “Towards isomorphism in development aid? A study on current trust patterns and their implications for the multi-actor policy on diversity in aid.”

Three Initial Assumptions About the Prevalence of OMD

Following from our initially formulated research question: “When, how and why does OMD occur?” – a first assumption was that we would indeed find many instances of OMD in the inter-organizational aid relations that we were about to study. Since we already knew, from previous joint studies and Janet's own extensive work experience in the field, that there is a substantial amount of regulation and numerous measurement technologies in this field, we initially also assumed that we would find OMD pretty much everywhere, or at least in most instances – albeit to varying degrees. We were hence particularly interested in the amount and frequency of regulation and measurement technologies employed by organizations in the donor role, where we initially assumed that these regulations and tools would help to explain the prevalence of OMD, essentially that: “the more regulations, the higher the prevalence of OMD.”

At first, we set out to study the degree of OMD in the four so-called actor groups who receive and channel Swedish public aid funding: (1) civil-society organizations (2) private-sector actors, (3) Swedish authorities in the public sector, and (4) research cooperation (see www.sida.se). ¹ Because the different actor groups have somewhat different financing mechanisms and criteria for how to apply for funding, we found it reasonable to believe that one or another group may be more regulated and hence more or less likely to suffer from OMD (see Carleson, 2017; Resare, 2011).

In order to compare how the actor groups' different institutional domains were regulated, we analyzed the overall governance mechanisms for the four groups, i.e., their separate aid allocations, strategies, agreement templates, performance measurement and other control requirements, formal and informal guidance documents, formal and informal organization structures, etc. We also conducted over 20 interviews with representatives from Sida and the Ministry for Foreign Affairs, and representatives or responsible managers for the respective actor group. In addition, we participated in a number of more general events, debates and discussions applicable to all actor groups (e.g., events organized by Sida's Forum on Working Adaptively and the Network for Learning – Capacity Development in Practice, or events organized for a specific actor group, e.g., a research day organized by International Science Program (ISP) for all research partner organizations).

In addition to the general macro studies used to study the first hypothesis (i.e., our first assumption), we selected case organizations to represent each of the four actor groups and conducted interviews with their representatives as well as studying their agreements, evaluations, performance measurement, and control requirements in detail. We looked at documentation such as contractual requirements including management technologies such as the “logical framework,” but also at additional requirements (such as random checks), and looked for situations where these artifacts or decisions (e.g., a decision to contract a third party, such as a consultant) played a key role in continued collaboration. Written documents like agreements, contracts, e-mail exchanges, and other project-related texts have also been valuable to the study as they offer an opportunity to follow and compare how formulations, stipulated conditions, sanctions, etc., are passed on unchanged – or rather, after being changed and/or extended by the organizations in the aid chain and its wider network.

We have studied the following seven case organizations and their respective aid relations: the International Science Program (ISP) – an example of support through universities, Union to Union – an example of support through civil society organizations; RFSU (the Swedish Association for Sexual Education) – an example of support through civil society organizations; the Swedish Environmental Protection Agency (SEPA) – an example of support through public-sector authorities; and Volvo – an example of support through private-sector actors. In a data collaboration with three graduate students working on two master's theses (Laurén, 2019; Smith & Ringqvist, 2019), we also had the opportunity to follow the cases of the Swedish Chemicals Agency – another example of support through Swedish authorities, and Forum Syd – an example of support through civil society. Both master's studies followed several aid-channeling organizations (from the aid organization based in Sweden to organizations operating in the developing countries), and posed questions regarding when, how, and why OMD occurs or not in these aid relations.

The second of our initial assumptions was that the prevalence of OMD would also increase as we tracked the contractual relations along the vertical “aid chain” from the Ministry of Foreign Affairs and Sida, down to the aid organizations in the different actor groups and their partners in the developing countries. Above all, we wanted to explore whether there was a tendency for aid organizations in the donor role to add even more control requirements before turning to the next organization in line, in the recipient role. We hence hypothesized that “the greater the number of relationships in the aid chain, and the further down the chain you come, the higher the prevalence of OMD.”

After concluding that the “aid chain” is in fact more of a web of aid relations (see also Chapter 7), we developed a third hypothesis, which was that the prevalence of experienced OMD may be explained by horizontal relations (such as those to external experts) in the “aid web,” based on our assumption that: “The more external experts that have sold measurement technologies to the aid organizations involved, the higher the prevalence of OMD.”

Thus we also studied the aid organizations' agreements and contracts with external experts and interviewed external experts on their views on when, how, and why OMD occurs. We interviewed the group of consultants at NIRAS (then InDevelop) who were working with the overall Sida framework agreement twice, in 2013 and in 2022, and in 2018 we interviewed the consultants from the AIMS consultancy company. We also conducted interviews with thematic Help Desks procured by Sida. The main method used to analyze the historical material was process tracing (Collier, 2011), where the researcher tries to unfold why a specific event or change happened by testing different assumptions. Here, we found that a changed demand for external expertise was linked to a changed perception of uncertainty.

Key Findings: Aid Organizations Are Similar and OMD an Experience

Our explorations based on the three initial hypotheses (described above) gave us some interesting tentative answers, but also spurred us to broaden the scope of our study, as described below.

In our studies of the first hypothesis (H1. the more regulations, the higher the prevalence of OMD), we found no clear pattern in the data to verify this common assumption. Rather, we came to conclude that OMD is not an objectively verifiable state, but an individual experience of a perceived overregulation (of an aid project, organization, inter-organizational relationship, or larger system). Moreover, and importantly, we conclude that it is not the amount of regulations per se that leads some people in the aid system to experience OMD. As discussed at length in several chapters, the prevalence of OMD, or rather the chances of preventing or counteracting such tendencies, depends instead on whether aid bureaucrats (and other people in the system) have the knowledge and courage to make sense of the regulation and oversight, and to interact with it in a meaningful way, using approaches and practices that we call “pragmatic bureaucracy” (see Chapter 8).

In some cases, we even found, counterintuitively, that increased oversight and control led to positive outcomes, such as increased trust and a sense of visibility for a specific topic or specific task carried out by someone in the organization (see Chapter 6). Based on these findings and analyses we conclude that there is not a straightforward answer to the question of whether reduced oversight on its own would result in less OMD experienced in aid (see Alexius & Vähämäki, 2020). Concerning the comparisons between the different types of organizations and the assumed differences between them, independent of their institutional context we have found that aid organizations and their aid bureaucrats are more alike than different, something we discuss in chapters 4, 5, and 7.

Our tentative findings concerning the second hypothesis (H2. the greater the number of relationships in the aid chain, and the further down the chain you come, the higher the prevalence of OMD) are discussed in Chapter 4 where, on one hand, we present data that supports the assumption and, on the other, data that speaks against it. In essence, we found that most aid organizations are expected to be “plural actors” who frequently switch between the two main roles in the core role set of development aid operations: the donor and the recipient. We conclude that each aid bureaucrat who takes on the donor role on behalf of their organization tends to be eager to meet the social expectation of being a responsible, “proper” donor. And by this is typically meant ensuring continued (and sometimes increased) regulation and oversight. Although more research is necessary, our tentative findings indicate that, although aid organizations and their bureaucrats switch between the roles of donor and recipient, the donor role may come to dominate the bureaucrats' approach to regulation and control. However, we also find attempts to balance the tendency to overregulate. Such initiatives are initiated in part by aid bureaucrats who act in the recipient role, and in part by external experts who may also advocate exceptions and adjustments to regulation and control schemes (see Chapters 5–7).

Considering the third initial hypothesis (H3. the more external experts that have sold measurement technologies to the aid organizations involved, the higher the prevalence of OMD), based on our tentative findings we do not rule this out. However, as described in Chapters 6 and 7, while some external experts appear to have increased the risk of OMD, others helped to counteract such tendencies.

Empirical Material and Analysis

In terms of empirical data, our study is based on the analysis of hundreds of documents and some 80 transcribed interviews with aid bureaucrats working at different levels in different organizations, including public agencies, private companies, NGOs, and universities, all of which are involved in development aid projects fully or partially financed by the Swedish taxpayer.

The semi-structured interviews were conducted by the authors in different projects dating back to 2013, with an emphasis on the period 2017–2023. We used a rather broad and open, thematic interview guide where we let the interviewees speak as freely as possible about key topics related to governance and management of aid. All interviews lasted between 1 and 2 hours, and we asked about perceptions of control, trust, and learning, and also more generally about the nature of the relationships to other organizations, in both the donor and the recipient role. There were questions about the specificities of the actor group to which the organization in question belongs, about its “intermediary” role as such, about popular management technologies, evaluations, contracts, and other governance mechanisms. As the study progressed, the interview guide was refined, for instance, to follow up early findings that indicated the importance of a “role-switching” and the balancing power of “pragmatic bureaucratic approaches,” but the topics and focus remained the same.

The over 140 hours of interview time have been transcribed verbatim by a certified transcription firm (approximately 40–50 pages for each interview). The documents, field notes, and verbatim transcriptions, along with our own reflections noted after each interview session, were then analyzed in several rounds and coded abductively, that is, combining insights from theory and the data at hand (Alvesson & Sköldberg, 2008). In terms of data validation strategies, access to both interviewees and other sources of verbatim accounts (such as contracts and information on websites) means that the case narrative is to a large extent based on so-called low inference descriptors, i.e., verbatim accounts of the interviewees' own views (Johnson, 1997). Since the project is a collaboration between two authors, all coding and analyses were first conducted individually by each author, after which another round of recoding and analysis was conducted jointly. In an interpretative mode, a cross-reading of all of the interview transcriptions and field notes was performed by the authors to tease out analytical patterns and relevant themes. Allowing time between the rounds of analysis provided an opportunity for further reflection, enabling the discovery of new meaningful patterns (Davies & Harré, 1990). Although all of the data collected was considered in our coding and analyses, only a selected number of sources are referred to in the narrative of the case, to illustrate patterns interpreted as representative for the material as a whole.

Translated into triangulation strategies for validating data (Johnson, 1997), we used (a) data triangulation where we analyzed verbatim transcripts of interviews, written documents, website data, and field notes from participant observations, (b) methods triangulation (document studies, interviews, participant observations), (c) investigator triangulation, in the sense of teamwork between two authors in collecting and interpreting the data, and last but not least (d) theory triangulation, where we have used multiple theories and perspectives to help interpret and make sense of the data (i.e., such as institutional theory on social roles and scripts, theory on trust formation and theory on governance under conditions of uncertainty and at a distance). None of the interviewees requested anonymity, and all gave their oral consent to use the interview data in academic publications.

Further Verification

In some cases, we have had the benefit of participant feedback as we have presented early drafts to interested interviewees and tentative findings in later follow-up interviews and in practitioner lectures. Participant feedback of this sort represents another validity-promoting strategy to verify interpretations and extended insight (Johnson, 1997). In our case, practitioner feedback has resulted in both minor revisions and valuable clarifications. A number of informal, non-recorded but informative conversations, face-to-face or via Zoom, have added further insight and nuance to the findings.

Since we are primarily interested in organizations and institutionalized roles rather than the roles of individual persons, we have deliberately chosen not to publish the names of those interviewed. Having stated this to the interviewees, we believe that this has further contributed to the open and frank accounts received. And although we have decided to publish the names of the organizations, we hope that the reader will find, as we have, that the similarities across the cases are more pronounced than differences, making the individual cases in themselves less paramount.

Along the research journey, earlier findings have been further verified via follow-up group interviews with aid bureaucrats, external experts, and other scholars (with the final sessions held in early 2023). Our analysis has also benefited from the insightful comments and advice received at several international academic conferences, and we wish to thank the conveners and participants of the following: Nordic Development Research conference, Gothenburg, 2018; Political Resource conference, Södertörn, 2019; European Group for Organization Studies (EGOS), Edinburgh, 2019; Organizing the World conference, Stockholm, 2022; the EGOS conference in Cagliari, 2023 and the Nordic Development Research conference in Uppsala, 2023. Furthermore, we have had continuous opportunities to present and discuss our findings with a dedicated reference group consisting of: Göran Sundström, Professor of Political Science at Stockholm University; Bino Catasús, Professor of Business Administration at Stockholm University; Joakim Molander, PhD, Head of Budget and Program Performance International IDEA in Stockholm; and Kristina Tamm Hallström, Associate Professor of Business Administration at the Stockholm School of Economics.

Throughout the research process, we have aimed for reflexivity in the sense of critical self-reflection on our potential biases and predispositions, where having two researchers to gather and analyze the data adds a valuable dimension (Otley & Berry, 1994). Employing these kinds of validating strategies in data generation and analysis has been argued to push the exploratory research toward both methodological and theoretical rigor (Eisenhardt, 1989; Stebbins, 2001). Last but not least, we wish to make clear to the reader that this volume makes no claims providing a fully representative picture of the operations of every aid bureaucrat and aid organization involved in the global field of development aid.