The impact of story: measuring the impact of story for organisational change

Introduction

The impact agenda is a highly contested space and has been criticised for limiting creativity and indeed changes in practice beyond academe (Alvesson and Sandberg, 2013; Johnston and Reeves, 2017). As such, rather than “in the box” thinking, there have been calls for “box changing, jumping or transcendence” for more imaginative approaches (Alvesson and Sandberg, 2014, p. 967) which engage stakeholders in collaborative forms of inquiry (Cunliffe and Scaratti, 2017; Ozanne et al., 2017; Pettigrew and Starkey, 2016; Wall, 2013, 2014, 2015, 2016a, b, 2017a, b, 2018).

Within this context, MacIntosh et al. (2017) highlighted the importance of dialogue and reflexivity, and the role of the importance of narrative within the impact debate. Alongside this, story-based and narrative approaches are gradually becoming more respected as an effective tool for learning and development and for understanding organisational change (McCormack and Milne, 2003; Gabriel, 2008; Gabriel and Connell, 2010; Reissner, 2011; Pässilä and Vince, 2016). Evidence of impact has included: service improvement in health care settings (IDEA, 2009; SCIE, 2010; Ellis et al., 2011); positive impacts on policy (in terms of client outcomes) (IDEA, 2009; Clark and Purdy, 2007; SROI Network, 2011); improvements in performance indicators (Schalock, 2001); improvements in staff engagement (MacLeod and Clarke, 2009); and improvements in well-being outcomes (Boorman, 2009; Rath and Harter, 2010; NEF, 2011).

However, although there is a diversity of potential methods and strategies to evaluate story-based interventions, there is no agreed standard or process. Therefore, a practice problem facing the practitioner researcher using story-based methods in the workplace is how to analyse, interpret and present the data in a systematic way that results in credible evidence. As Guest et al. (2012) propose “good data analysis (and research design, for that matter) combines appropriate elements and techniques from across traditions and epistemological perspectives”. In this way, evaluation can not only evidence the project outcomes but also create convincing links to personal learning as well as wider organisational development objectives, thus adding credibility to story-based methods.

This paper draws from a practitioner research project in the UK, as part of a work applied learning and organisational development project, to evaluate the impacts of two case studies. In order to achieve this, however, the practitioner researcher had to develop an appropriate evaluation framework and methodology which was ecologically appropriate for the well-being and narrative nature of the project, the practice setting of the practitioner researcher and generated valid results which could then be utilised in practice to support organisational development and service improvement.

This paper is structured as follows. The first section reviews some of the key evaluative methods and tools which are used in practice to measure impact and organisational learning in the context of health and social care organisations. The second section then outlines the methodology adopted as part of this study, exploring the suitability of various evaluative methods in the context of health and social care settings. The following sections then present and compare two case studies, highlighting the key impacts and broader findings from the case studies. Finally, the paper moves to a discussion of some of the challenges of evaluating story-based methods for organisational learning and change, and reflects on the stages of designing robust evaluative frameworks in the context of story and health.

Assessing impact in health care

Over a decade ago, a cross-government and social care sector working party produced the document “Putting People First: Transforming Adult Social Care” (IDEA, 2009) setting out the vision for adult social care and its direction over the next ten years. This paper was a keystone paper as it set forth a strategic direction which is generically known as “personalisation”, or highlighting the importance of the individual experience. Similarly, Shepherd et al. (2010), in their position paper for the Sainsbury Centre for Mental Health, identified peoples’ lived experience as the most potent driver of organisational change within a culture of recovery. This has positioned and framed the work of external providers ever since, with an emphasis on co-production, laying the ground for participatory methods of working and of evaluation.

In terms of approaches to evaluating work within this broader professional context, there are different varieties to how and why evaluation is done. For example, Trochim (2006) postulates that evaluation strategies fall broadly into four major groups: scientific/experimental, management-orientated systems, qualitative/anthropological and participant-orientated (the latter of which seem appropriately aligned to the context). In contrast, Mertens and Wilson (2012) propose four categories of evaluative purpose: to determine inputs and need, to improve or change practices, to assess programme effectiveness and to address issues of social justice. Again, these seem relevant to helping decide the frame of practitioner oriented evaluation in the above professional context.

Within these broader approaches, there are specific methodologies which are used in contemporary health care settings. One of the most popular, and which continues to influence many other models, is Kirkpatrick’s (1998) model and toolkit, which was developed as an evaluation tool for assessing impact and outcomes of learning and development programmes. Bespoke methodological approaches utilising Kirkpatrick’s thinking have been developed by governments. For example, The Impact Evaluation Model uses principles of outcomes-based accountability, and has been recommended by the UK Government for localised impact evaluation of activities especially around service and workforce reform. Reio et al. (2017), however, critique Kirkpatrick’s work as being overly focussed on the achievement of outcomes of training rather than on the impact on the stakeholder and whether their needs have been met. Reio et al. propose that stakeholders should be able to input to design, development and evaluation.

Return on investment (ROI) models have also been adopted to measure impact in a very specific and narrow sense (Wall et al., 2016, 2017). More recently, social return on investment (SROI) methodologies have also appeared which have also been participatory by nature, and emphasise those outcomes which are valued by people, including stakeholders and beneficiaries of social programmes, and provide a participatory mechanism for their voice or story to be heard. For example, The SROI Network (2011), which promotes the use of SROI methods internationally to address social injustice, claims:

SROI tells the story of how change is being created by measuring social, environmental and economic outcomes […] SROI is a framework to structure thinking and understanding. It’s a story not a number. The story should show how you understand the value created, manage it and can prove it

(SROI Network 2011, p. 2).

Other forms of participatory evaluation methodologies typically assess progress, performance and impact of a project, but with a primary objective of creating a culture of learning for project staff, beneficiaries and partners. Hasenfeld et al. (2004), as an example, promote the participatory model of evaluation (PME) as a highly collaborative process, relying upon a feedback loop from partners and staff. In their work, Hasenfeld et al. (2004) explored how involving clients in the community in ongoing feedback makes them part of the evaluation process. The validity accorded to case studies by PME lends credence to personal narratives as a methodology in evaluation.

The practical issues of implementing such complex evaluation approaches can stifle widespread use (Wall et al., 2017). In contrast to complex methodologies, Davies and Dart (2005) claim that the most significant change (MSC) technique serves as a legitimate form of participatory monitoring and evaluation. MSC was first developed as a means of auditing changes in overseas development aid projects, but can support organisational learning and service improvement. It is participatory because of the multiple perspectives elicited. As Davies and Dart (2005) explain:

[…] it contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole […] MSC makes use of […] “thick, description”, closely textured accounts of events, placed in their local context, and where the role of the observer and their subjectivity, is visible. In the world of ordinary people these often take the form of stories or anecdotes

(p. 67).

Methodology

This paper adopts a case study approach to document and examine the impact of story in the context of health care organisations, and was undertaken by a practitioner research seeking the dual roles of contributing to the development of the organisations and generating new practitioner knowledge for the individual (Wall, 2014; Heikkinen et al., 2016). The two case studies relate to two story-based intervention projects focussed on organisational development and service improvement as dual outcomes. The projects were delivered within two public sector organisations: one is an adult social services organisation (now referred to as “Social Care Co.”) and the other is a health care organisation (now referred to as “Recovery Co.”) in England.

The intention was a form of case study which was discovery led and inclined towards emphasising social processes and relationships within a natural phenomenon, rather than restricting the attention upon outcomes, and is also suitable for comparison case studies (e.g. of individuals or organisations). In this way, the descriptive case studies focus on contemporary events, explored in their real-life contexts rather than in controlled environment (Yin). The use of multiple cases also provides the opportunity to compare and contrast the findings across different real-life contexts, in terms of different real-life organisational cultures and story interventions (Yin, 2013). However, it is acknowledged that the case study approach is also vulnerable to criticism re-credibility of generalisations from findings (Denscombe, 2010).

Several options were considered when designing the project for suitable data collection methods. However, given the nature of the projects, it was argued that evaluation can be a “sense-making process” in organisations (Weick et al., 2005; Weick, 2016), as well as one that collects and interprets data, and sharing of personal stories could be a useful experience for participants in the evaluation. Furthermore, it was also argued that practitioner researchers in the context of providing services to health care organisations need to consider how the provider-client relationship might be affected by their choice of methods, for example, a rigorous “root and branch” investigative survey might jeopardise future relationships.

It was therefore decided that the project data would be collected through semi-structured interviews incorporating the MSC method (Davies and Dart, 2005). This was chosen as it was the most ecologically appropriate for the well-being and narrative nature of the project, the practice setting of the practitioner researcher and generated valid results which could then be utilised in practice to support organisational development and service improvement. The interview guide, which was the initial proposed evaluation framework to be used with story projects, is presented in Box 1. For both case studies, purposive sampling (or purposeful sampling) was used for data collection, with between 6 and 12 staff and service users. The evaluation framework (interview questions) was initially trialled outside of the two evaluations and questions which appeared to prompt repeated answered were adjusted.

Case study 1: Recovery Co.

Case study 2: Social Care Co.

Discussion

A cross-case analysis of the findings of both projects indicated similarities around dimensions: how story work underpins radical organisational cultural change, its training application for staff to be better educated around new policies and approaches in health and social care, and its impact on professional relationships particularly partnership working and with service users. A strong indication from this study is that story work enhances team building and benefits new projects in the early stages, as strong organisational outcomes were demonstrated for both projects.

The benefits to Recovery Co. were significant enough for both strategic level and other staff to extrapolate ways of integrating story work into management practice, such as staff support, knowledge sharing and leadership development. In as much as story work evidences good practice and aligns with transformation of services, both projects stated that the outcomes of the story projects potentially enhanced the reputation of the organisation as an “honest broker” (Social Care Co.) or as “innovative” organisation (Recovery Co.).

Yet both projects were different in their focus and ongoing issues. The Social Care Co. project had a skills development focus to support the evidencing of personalisation, whereas the Recovery Co. project focussed on team building, culture change (towards a “recovery culture”). In the Social Care Co. project, participants further reflected in broad terms on sustainability and developing systems to support their “story gatherers”, whereas in the Recovery Co. project, the reflection was towards further exploration of narrative approaches and how these could improve best practice at all levels.

Key themes and outcomes from the interviews were therefore mapped visually for each project using wordle software. Wordles are easily created from key words emerging from the data as visual images; words are “weighted” by occurrence, represented as the larger words in the wordle. These were shared with the clients as a thematic illustration of project outcomes to assist with personal and organisational learning. The Social Care Co. wordle highlights that improvement in skills was dominant (story writing skills, creative thinking, better listening) as well as improvement in service-related relationships (partnership working, engagement, personalisation) (see Figure 1), whereas the Recovery Co. project wordle reflects the current recovery team’s focus on change and on therapeutic relationships (change management, relationships, culture change, well-being) (see Figure 2).

There are also wider implications of such variability in project impacts and outcomes. Specifically, it was recognised that some of the evaluation framework prompts were not necessarily relevant in both contexts, and reflected the nature of the original scoping of the project (as discussed above). The initial evaluation framework that was developed for the purposes of evaluating story work in workplaces therefore needed to be adjusted to reflect the diversity of projects that would be developed. Reflections and decisions about this are reflected in Table III.

This reflects the responsive design of evaluation in workplace learning projects. For example, on consideration, questions 2 to 4 in the evaluation framework are most relevant where the project involves skills training or/and mentoring, and less so where the project delivers service user interventions or team-building workshops. Questions 6 and 7 are difficult to answer if the participants are not responsible for or knowledgeable of strategic and organisational goals, or where projects involve participation by stakeholder and partnership organisations. Similarly question 9 presupposes that the project is delivered to those who have direct relationship with service users. As such the evaluation framework design needs addressing early into the project design, and purposively linked to organisational outcomes – and reflects Reio et al. (2017) critique of evaluation being overly focussed on the achievement of outcomes of training rather than on the impact on the stakeholder and whether their needs have been met.

Conclusion and implications

This paper concludes that narrative or story-based work is efficacious and credible in generating workplace impacts, especially in the context of service transformation and improvement, and that practitioners can examine such dimensions in participatory ways. The willingness of staff to be involved in the project that this paper has examined further demonstrated that evaluation is regarded as valuable and a way, in itself, of engaging staff. Significantly, the involvement of service users in the evaluation was also said to have “recovery potential”, which further emphasises the suitability of participatory methods of project design and evaluation as well as research more broadly (IDEA, 2009; MacIntosh et al., 2017).

The richness of the evaluation reflect two areas: the reported processes that story activates and shapes, including sense making, team working, re-framing and collective empathy towards workplace impacts (Gabriel and Connell, 2010; Reissner, 2011; Wall and Rossetti, 2013; Pässilä and Vince, 2016), but also the reported impacts of practical and participatory forms of MSC-informed evaluation processes which also facilitate similar processes of sense making, framing, re-framing and collective empathy, but also motivation to act, change and improve (Wall et al., 2017). In addition, the elicited experiential content generated through the evaluation was found to be persuasive when presenting to higher level managers, as it provided strong links between the story work and organisational strategic priorities and pathways. As a result, there are a number of specific implications for different stakeholder groups, and these are represented in Table IV.

In this way, this paper argues that evaluative frameworks benefit from being designed in conjunction with the client organisation to align with their outcomes and be conducted through participatory forms. The decision to adapt the “MSC” method and integrate this into the evaluation framework enabled the strong links to the use of stories as data and evidence. Moreover, the MSC domains of change can be identified by a top-down or bottom-up process, through participatory consultation – in other words – the framework can be adapted to the specific aims and cultural context of the project, for example, more or less skills content, more or less service user involvement.

Findings showed that the MSC-informed questions can generate important stories as data in work-based projects, and can accommodate scaling up. In addition, participatory or co-production of evaluative design has exciting potential, and one which aligns readily with guiding ethos within health and social care organisational governance and culture. In this way, this paper documents contemporary evidence of the variety of organisational development and service improvement that story work can generate as part of workplace learning projects.