Disaster events are a ‘condition or situation of significant destruction, disruption, and/or distress to the community’ (Commonwealth of Australia 1998, ix). Such events can have severe, long term social, economic and environmental impacts. In New Zealand, the 2010 and 2011 Canterbury earthquakes were estimated to have a financial cost of approximately $15 billion, and reduced total Gross Domestic Product (GDP) in 2011 by around 1.5% (NZ Treasury 2011, 96). In Australia since 2009, natural disasters have claimed more than 200 lives, destroyed 2670 houses and damaged a further 7680, and affected the lives and livelihoods of hundreds of thousands of Australians (Productivity Commission 2014, 3). Human-caused disasters, such as acts of terrorism, nuclear accidents, anthropogenic fires, and transport incidents, have similarly devastating impacts. For example, the World Health Organisation’s (WHO) extensive study of the health impacts of the 1986 Chernobyl nuclear disaster concluded that the accident was responsible for long-term physical health problems and deaths from radiation exposure, and had significant impacts on mental health and wellbeing of the general population (Bennett et al. 2006, 69-96).
Disaster or emergency management aims to reduce the short and long-term impact of a disaster event. It includes pre-disaster interventions to reduce the potential future impact and help community preparedness, immediate response and relief efforts, and ‘post-disaster recovery’ that commonly refers to the period of time and activities that occur after the immediate relief and response to a disaster event (Commonwealth of Australia 2011, 29; FEMA 2011, 8; Ministry of Civil Defence & Management n.d.). Effective post-disaster recovery is critical to getting community members ‘back on their feet’. However, Archer et al. (2015) concluded, based on a review of post-disaster literature, that this phase of disaster management is poorly defined. This paper will focus on post-disaster recovery as it is a critical phase of the disaster management process in need of greater clarity.
Post-disaster recovery is highly complex. It occurs in an environment of high stress, involves multiple agencies and stakeholders, has multiple priorities that evolve over time, and has no clearly demarcated end point (Commonwealth of Australia 2011, 3–6). Government intervention is generally required to assist the affected community during recovery. In Australia and New Zealand, each tier of government provides assistance to support recovery.
In New Zealand, the legislative framework for emergency management is provided by the Civil Defence and Emergency Management ACT 2002 (CDEM). The Ministry for Civil Defence and Emergency Management is mandated to provide overarching guidance for post-disaster recovery. The framework, presented in Focus on Recovery: A Holistic Framework for Recovery in New Zealand, outlines the roles of different stakeholder groups. Consistent with the approach adopted in Australia, the community is at the center of recovery, with government and other stakeholder groups providing support and assistance. CDEM Groups – comprising local authorities working in partnership with emergency services and major utilities – have a coordinating role, and lead the development of recovery plans. These are enacted by government departments (with particular focus on local authorities) and non-government organisations (Ministry for Civil Defence & Emergency Management 2005, 3, 18–19)
In Australia, the federal government provides funding and support to state and local governments as well as to businesses and the community (Productivity Commission 2014, 8). State and territory government agencies are directly involved in the on-ground response and recovery effort, and include a disaster management group that provides overarching coordination and direction during and after a disaster event. Local governments play a key role throughout the disaster recovery process, and support the community over the long-term. In addition to the three tiers of government, there may also be a regional or district disaster management group that plays a coordinating role (Commonwealth of Australia 2011, 4–5).
Public expenditure on post-disaster recovery is significant. Over the past decade, the Australian Government alone has spent around $8 billion on post-disaster relief and recovery (Productivity Commission 2014, 3) and forward estimates indicate another $5.7 billion is expected to be spent on past disaster events (Australian Audit Office 2015, 132). The New Zealand government is estimated to have contributed $15.2 billion to recovery from the Christchurch earthquake, and forward estimates predict a total of $40 billion will be spent on the rebuilding effort (The Treasury 2013).
Given the significant amount of public expenditure and the importance of the post-disaster recovery phase, evaluation can be useful to ensure that resources are efficiently allocated to achieve effective outcomes. Evaluations are critical to facilitate learning and continued improvements to the post-disaster recovery process to achieve desired outcomes. A recent report on Evaluation in government by the UK National Audit Office identifies a key purpose of ex-post evaluations (i.e. evaluations undertaken after policy implementation) as ‘a means to improve existing policies and to better design future policies’ (National Audit Office 2013, 5). Further, as Brecher et al. (2005) note in their exploration of expenditure analysis as an evaluation tool, evaluators of long-term, complex interventions (of which post-disaster recovery is undeniably one) are often expected to provide interim feedback to determine whether a project is ‘on-track’. The findings from these interim evaluations are used to guide decision making on whether revisions are required for the project, or if indeed it should be continued at all.
Undertaking evaluation can cover a variety of questions and stages of a project. While many definitions of evaluation are used, the term generally encompasses the systematic collection and analysis of information to make judgments, usually about the effectiveness, efficiency and/or appropriateness of an activity (The Sphere Project 2015; Australasian Evaluation Society 2010; Owen 2006; Ryan 2014). Effectiveness refers to the ability of the program or activity to achieve the desired goals (i.e. did it work?), efficiency considers whether resources are being used wisely (i.e. the relationship between inputs and outputs), and appropriateness examines whether the program or intervention is suitable for meeting its objectives in the policy context (i.e. was it the right intervention for the need or stated problem?). Figure 1 outlines the types of questions that relate to the key evaluation themes of process, appropriateness, efficiency, and effectiveness.
The logic of a program and the relationship with key evaluation themes
Source: Ryan (2014)
Despite the potential value of evaluation, there is currently no existing national framework for monitoring or evaluating post-disaster recovery in either New Zealand or Australia. This is a significant barrier to jurisdictional learning from previous post-disaster experience, which would help to improve the effectiveness and efficiency of government interventions (at all levels) as well as provide rationale for program design and government investments. A recent review of Australian emergency management evaluations by Dufty (2013) demonstrated the current inconsistencies in post-disaster recovery evaluation. This brief review of a sample of evaluations found that often no evaluation was conducted at all. If they were conducted, the type and timing of the evaluation were highly variable. Similarly, the review by Archer et al. (2015) found that there was no common understanding, definition or indicators of success for post-disaster recovery. There has yet to be a systematic review of the post-disaster evaluations in New Zealand. However, without a national monitoring and evaluation framework, we anticipate there would be similar variations in the conduct of evaluations, as well as their type and timing. With the development of a framework outlining definitions, indicators of success and useful evaluation measures, evaluation practice would improve and increased learnings about effective interventions for post-disaster recovery could be fostered. Consistency in monitoring and evaluation of post-disaster recovery interventions would provide further guidance for practitioners in the field, and the basis on which to build an ongoing knowledge base.
This paper reviews existing evaluations of post-disaster recovery. It extends the initial review undertaken by Dufty (2013) and conducts a more extensive, systematic search and analysis of existing post-disaster evaluations from the Australian, New Zealand and international grey literature. The purpose of the review is to gain further insights into the extent and type of evaluations of post-disaster recovery interventions and the methods used in practice. To achieve this purpose, the type of interventions included in post-disaster recovery is investigated. This will contribute to understanding of the post-disaster recovery phase, and help to clarify it.
The review will identify trends in the methods used for evaluations, and aims to assist program managers to identify appropriate and comparable case studies to help guide evaluation of post-disaster recovery efforts. The findings of the review will be useful for informing the development of a post-disaster evaluation framework that will improve the application and use of evaluation. As disasters potentially impact all locations, and recovery commonly involves each level of government, the outcomes of this review are relevant to disaster recovery program managers across New Zealand, Australia, and internationally.
The search method
To identify evaluations of post-disaster recovery interventions, the literature search focused on operational material and other grey literature. We used this approach in order to distinguish evaluations conducted for academic purposes from empirical or practical evaluations that were done to assess the performance of a program or initiative. We conducted a search of the grey literature from national and international disasters identified from the Australian Emergency Management Knowledge Hub1 (the Hub). The Hub provides details of 680 disasters from Australia, New Zealand, and internationally. Dating back to 1791, it includes materials from natural and human caused disasters such as bushfires, cyclones, floods, shipwrecks, criminal activities (including acts of terrorism), and epidemics. To narrow the search the response was limited to disasters that occurred since 1995. We made the assumption that grey literature may be difficult to access online for events that occurred before this date. The focus here is on evaluations of post-disaster recovery for disasters that occurred before January 2015. Post-disaster recovery is an extended process that can take a number of years, and evaluations are thus unlikely to occur immediately after the event (Labadie 2008). This reduced the result to 217 disaster events. The database includes all disaster types. For this review, the focus was on:
The choice of these disaster types reflects the all hazards approach to disaster management that is being adopted in Australia, New Zealand, and internationally. The all hazards approach to disaster management refers to the ability for arrangements and programs to deal with a wide variety of hazards, including natural and human caused disasters (Commonwealth of Australia 2011, 147). The disaster types that are the focus of this review included those that were most common according to the data provided on the Hub.
Using the above criteria, we identified 73 disasters. The Hub is the only catalogue of disasters for Australia and New Zealand, and it includes international examples. However, there were some notable disaster events that were not identified in the search. In particular, there was under-representation of disaster events outside Australia. To augment the findings from the Hub, we added an extra 11 prominent disaster events to the list. Prominent disasters were identified as those that received significant media attention, and were chosen through a Google search. The resulting list of disasters that were the focus of the grey literature search was 84.
To search the grey literature for existing evaluations for the identified disaster events, we searched the resource section of the Hub and the relevant government websites, and also conducted a general web search. The search terms included the name, location, type and date of the disaster as well as the terms recovery and evaluation/review/outcomes/success. We recorded and analysed the evaluations identified from the search using the typology presented in the following section.
A typology to categorise disaster events and evaluations
We developed a typology to review and analyse the grey literature identified through the search. The typology provides a useful framework for identifying case studies of evaluations. It can be used to inform the development of a post-disaster recovery evaluation framework and to assist in its practical application. A case study is a research strategy (Yin 1981) that enables ‘detailed contextual analysis of a limited number of events or conditions and their relationships’ (Dooley 2002, 335). It focuses on just one or a few instances of the phenomena being researched, and allows for an in-depth study. The case study draws on a range of methods of data collection, and uses triangulation of the evidence to investigate theoretical or stakeholder propositions (Yin 2003).
A draft typology was created prior to reviewing the identified grey literature. This included the basic elements to categorise a disaster (location, disaster type, date of occurrence and estimated impact), whether or not an evaluation was undertaken, and the evaluation methods. The typology was revised and amended as additional elements emerged that were informative for categorising the disaster events, may potentially impact on evaluation, and assisted in the analysis and categorising of the evaluation methods. Table 1 presents the typology and description of each category.
A typology for post-disaster recovery evaluations
Disaster type, location and date of occurrence
The characteristics of the disaster were recorded including type (e.g. cyclone, bushfire etc), location and date of occurrence. This information identifies the individual events and any trends between these characteristics and the evaluations conducted.
Indicators of extent of impact
The impact of the disaster affects the recovery effort. The estimated impact was difficult to identify from the grey literature as there are a number of different ways in which this can be assessed. The scope was narrowed down to the immediate social impact represented by loss of life, and the estimated economic impact as the insured cost. The actual impact will be much more complex, however, further investigating this is beyond the scope of this review. The insured cost for the disaster events was provided on the Hub and normalised for current cost estimates. For additional disasters not listed on the Hub, the insured cost was reported where it could be found from the grey literature.
Post-disaster recovery effort
Types of activities/programs undertaken
The activities and programs for post-disaster recovery are extensive. The Australian Government’s Community Recovery Handbook 2 promotes a holistic approach to post-disaster recovery that covers social, economic, built and environmental dimensions.
Community involvement in post-disaster recovery planning
There is general agreement in the literature that community involvement in the post-disaster recovery planning process is key to achieving good outcomes, and it is one of the principles of post-disaster recovery outlined by the Australian Government (Commonwealth of Australia 2011).
Agencies involved in recovery
The policy context for post-disaster recovery is complex and frequently involves multiple agencies. In Australia, this includes local governments, state or territory government departments, the federal government, and NGOs and volunteer organisations. In New Zealand, local and regional authorities commonly take the lead to support communities and are assisted by the federal government, NGOs and volunteer organisations Internationally, post-disaster recovery similarly involves a multi-level governmental response. International organisations and support from other countries may also occur.
Post-disaster recovery evaluation
Number of evaluations conducted (if any)
The number of evaluation documents identified from the literature search.
As identified by Dufty (2013), there are different types of post-disaster recovery evaluations. These include government inquiries, independent evaluations and operational reviews.
Who undertook the evaluation
The party that undertook the evaluation was recorded. This may include a government department, external consultancy, independent researcher or NGO.
Process or outcomes focused evaluation
Evaluations are commonly process focused or outcomes focused.
Process evaluations look at program implementation or resource allocation, whereas outcomes focused evaluations assess the impact of the activity or intervention.
Data collection methods
The method for data collection to inform the evaluation.
Focus of the evaluation
The aspects of post-disaster recovery that the evaluation focused on.
This was further categorised into social, economic, built or environmental domains.
Bibliographic details of the evaluation.
The publication details of the evaluation document.
To present the data, the typology was separated into two tables2 (Appendix 1). Table 4 presents the disaster characteristics and the evaluations that were identified by the grey literature search. Table 5 provides further details on the evaluations conducted and the information source. Of the 84 disasters identified for the review, post-disaster recovery evaluations were only found for 35 disasters. For the sake of brevity, we include in the tables only those disasters where an evaluation of post-disaster recovery was identified.
Disaster characteristics and post-disaster recovery actions
Disaster events included in the review (summarised in Tables 2 and 3) occurred in Australia, New Zealand, USA, Canada, UK, Switzerland, Japan, Indonesia and the Pacific Islands. The nature of the search resulted in a strong emphasis on Australian disaster events as the Hub is an Australian based catalogue of disaster events. Bushfires and floods were the most common disaster type identified, with only eight criminal acts/terrorist events, three earthquakes, and two tsunamis reviewed. The scale of the impacts ranged from relatively minor (for example the bushfire in Coonabarabran, NSW in 2013) to devastatingly high (for example Hurricane Katrina in New Orleans). However, it is worth noting again that these are only crude indicators of the impacts of a disaster. The actual impact will be much more complex, and investigating this in further detail is beyond the scope of this review.
Outcome indicators for post-disaster recovery
Central city activity
Quality of life
Central city repair and rebuild
Horizontal infrastructure repair
Ease of travel and transportation
Drinking water sources
Drivers of the economy
State funding for higher education
Median household income by race and ethnicity
Educational attainment by race/ethnicity and sex
Jail incarceration rates
Size of city’s middle class
Size of city’s middle class by race and ethnicity
Suburbanization of poverty
Quality of life
Arts and culture
High school cohort graduation rates
Commuting by public transit
Most post-disaster recovery efforts addressed the social, built, economic and environmental domains. The types of recovery activities included:
Social: physical and psychosocial support such as healthcare, counselling and programs targeted to increasing community welfare such as art initiatives or memorials.
Economic: support to buffer and improve the local economy. This may include stimulus activities, assistance to primary industries or tourism, employment programs, or business counselling development.
Built: rebuilding physical infrastructure including housing, roads, bridges and other development. May also include re-zoning or relocation of residents.
Environment: restoring environments affected by the disaster. May include revegetation, monitoring and clearing waterways, stabilising coastal zones, or other activities to assist impacted ecosystems.
Government agencies and practitioners are adopting a holistic approach to recovery. In Australia and New Zealand, disaster recovery frameworks have been established to guide the development of post-disaster recovery programs. In Australia, the Community Recovery Handbook 2, released by the Australian Government Attorney General’s Department, states that community recovery should be coordinated across social, built, economic, and environment domains (Commonwealth of Australia 2011, 78). In New Zealand, the framework for recovery established by the New Zealand Ministry of Civil Defence and Emergency Management (2005, 6) similarly presents recovery as an integration of social, built, economic, and environment domains to support the community.
For some disasters, particularly acts of terrorism or criminal activity, recovery efforts focused on social and built domains. For example, recovery efforts for the Oklahoma City bombing in 1995 focused on rebuilding the affected area and providing services to address physical and mental health (Oklahoma Department of Civil Emergency Management n.d.). Similarly, recovery activities following the 9/11 terrorist attacks focused on social, built and economic aspects (US Government 2006). In these examples, not addressing environmental dimensions was appropriate. This highlights a potential issue with the all hazards approach to disaster management. Although a holistic approach for disaster recovery could be appropriate for most disaster events, it may not be appropriate for all. Similarly a national framework for post-disaster recovery evaluation will need to take into account the variability of disaster events and post-disaster recovery actions. While overarching frameworks and guidelines are useful for guiding practice, they should permit tailoring of activities or evaluations to the specific context.
The involvement of the community was variable in post-disaster recovery. Some post-disaster recovery efforts involved significant community involvement. For example, the community was heavily involved in planning disaster recovery following the Blue Mountains bushfire in 2013. Community meetings as well as online forums and surveys were used to identify community priorities (NSW Government 2013). For other disaster events, there appeared to be less focus on the community. This does not necessarily mean that community consultation did not occur. It may be that this aspect of the disaster recovery was simply not well documented. This would be expected as documentation in general for some disaster events was low.
Nonetheless, community involvement should be a high priority for post-disaster recovery. The importance of a community focused approach has been recognised in the disaster recovery frameworks for New Zealand and Australia, where community lies at the core of recovery, and the recovery principles state that the process should be ‘community led’ (Commonwealth of Australia 2011, 78; Ministry of Civil Defence & Emergency Management 2005, 6). The limited inclusion of community perspectives in some cases suggests that a gap exists between the theory of post-disaster recovery promoted at a national level and what is occurring on the ground.
Leadbeater (2013) argues strongly for the necessity of disaster recovery being community focused and tailored to community needs. In particular she highlights the importance of integrating appropriate community leaders into the planning process so that it is tailored to the existing values, networks, projects, relationships, knowledge, and capacity of the community. Similarly, Hawkins and Maurer (2010) argue that existing social capital in the community was key in assisting families to recover following Hurricane Katrina. The disaster events identified by the review that demonstrated high levels of community involvement may provide useful examples or guides for practitioners developing a community engagement strategy for disaster recovery.
A need for more consistent evaluations of post-disaster recovery
Consistent with the findings of Dufty (2013), our more extensive grey literature review demonstrates that evaluations of post-disaster recovery are often not undertaken in practice. Of the 84 disasters identified in the review, evaluations were identified for only 35 disaster events. The remaining 49 disaster events either did not have an evaluation, or it was not published or made publicly available. Given that it is common practice for government agencies to release these reports, we predict that the former is much more likely the case.
The lack of evaluations is a significant concern for the advancement of post-disaster recovery efforts and ensuring resources are efficiently allocated to achieve good outcomes. An evaluation determines how well a government program or intervention has met its objectives, holds officials accountable for its implementation, and provides insights for future policy making, including whether resources should be continued, increased or reduced (Althaus et al. 2007, 179).
National guidelines and frameworks for post-disaster recovery emphasise that community should be the focus for programs, with interventions intended to support communities in their recovery. However, the lack of evaluations means we do not know whether the community is actually benefiting from interventions. Without examining the appropriateness, effectiveness and efficiency of interventions, we cannot determine whether the programs being implemented are contributing to recovery and delivering outcomes. As such, it is unknown if the current spending on post-disaster recovery is justified and the most efficient use of resources, or whether funding and intervention approaches require a re-think.
Insufficient monitoring and evaluation is not uncommon for government interventions. The UK National Audit Office and the Public Accounts Committee (2013, 6) have criticised government evaluations due to:
‘gaps in the coverage of evaluation evidence;
insufficient use of evaluation evidence; and
difficulties faced by independent researchers in accessing administrative data and other government data to conduct their own evaluations of government interventions.’
There are a number of reasons why evaluations may be limited. Insufficient resourcing is a commonly identified barrier to evaluating a government program or intervention (DeLuca et al. 2010). Within a limited resource environment, priority is given to the implementation of the program. Unclear definitions and confusion over what is being evaluated also present a significant barrier. In recent discussions with disaster recovery personnel across Australia, it became apparent that there is a lack of understanding about:
what defines ‘post-disaster recovery’;
the steps or interventions that are included in this phase of disaster management, and;
what ‘success’ may look like.
This is further complicated by no clear end-point for disaster recovery intervention. Without these aspects being clearly defined, it is extremely challenging for practitioners to design and undertake evaluations. A national monitoring and evaluation framework should go some way to addressing these issues.
Disaster events identified by the review for which an evaluation was undertaken were often more recent, had a large social or economic impact, or were in more populated areas. The finding that evaluations were more common for recent events may reflect an increased emphasis on government transparency in the last 10–15 years. Governments are increasingly required to be more accountable for the resources being spent on implementing programs. As noted by Bovens (2007, 182), accountability of administrative agencies is the ‘hallmark of modern democratic governance’. Shkabatur (2012, 82) defines accountability as consisting of two elements: ‘the explanation and justification of agencies’ activities to the public; and an accompanying mechanism for public sanctions’. Evaluations address the first of these elements and, as outlined by Chouinard (2013, 238), are often defined as a ‘neutral instrument providing impartial, evidence-based, and objective information intended primarily to satisfy accountability requirements’. The notion that evaluations are becoming more frequent due to increased pressure on governments to be accountable for public spending is consistent with the association between the scale of impact and presence of evaluations. Those that have required greater resource investment may be more held to account for this resource investment.
Process versus outcomes evaluations
The review of academic disaster literature by Archer et al. (2015) found a strong emphasis on evaluating the effectiveness of the recovery process, as opposed to the impact and outcomes of the process. The following provides an example of the difference between outcomes and process focused evaluations.
A process evaluation looks at the actual development and implementation of a particular program. It establishes whether a numeric target has been reached and strategies implemented as planned. For example a process evaluation might confirm that 400 people applied for housing support after a flooding and that 350 were granted the support.
An outcomes focused evaluation is a systematic approach that measures the impacts, benefits, or changes that have occurred as a result of a particular program. For example a process evaluation can provide the quantifiable number of job reskilling that occurred following the earthquake devastation of a rural town’s only factory. The outcomes approach would tell you how many of those demonstrated increased confidence, changed behaviours, found jobs because of the new skills, etc.
The few existing evaluations identified by the review were mostly process evaluations (Table 4). Process evaluations seek to involve stakeholders in considering how activities occurred. The evaluation method seeks their views usually through focus groups, interviews or workshops. Examples include the after-action review undertaken for the bushfire in Coonabaraban, NSW in 2013 (Warrumbungle Shire Council 2013) and the government review of post-disaster recovery from the 2013 Tasmanian bushfires (Tasmanian Bushfire Inquiry 2013).
Evaluations that considered outcomes were generally undertaken by independent researchers or external consultancies. The outcomes evaluations identified by this review typically focused on the effectiveness of an intervention in enabling change by measuring a specific variable. This approach provides a data snap-shot, which while potentially useful for examining change over time, fails to connect program interventions with impact and outcomes. Recovery processes, outcomes and impact are inextricably linked. To determine whether specific programs are actually helping the community to recover, outcomes evaluations need to demonstrate the causal relationship between the intervention (and government spend), impact, and outcome. A sole focus on indicators without linking to the program or interventions is a barrier to researchers and evaluators making in-depth insights, with any judgments of success limited to the entire recovery program, or entire domains.
The snap-shot evaluations identified from our literature review represent an ‘outcomes only’ evaluation rather than ‘outcomes focused’ evaluation. They provide examples of useful indicators that may help inform outcomes focused evaluation, but further work is required to establish a causal link with interventions and impacts.
Evaluations that did consider outcomes tended to only focus on a specific aspect of post-disaster recovery – in particular, social and economic outcomes measured as mental health or wellbeing and economic activity (e.g. GDP, tourism income, industry profit). For example, the Mount Sinai Medical Center (2011) conducted long-term clinical assessments of physical and mental health to evaluate the effectiveness of support programs for those affected by the 9/11 terrorism event. The Regional Australia Institute (2013) examined the effectiveness of government recovery programs following Cyclone Yasi in 2011 using indicators of population growth, return of the tourism trade, performance of the agricultural sector, and employment levels. Frankenberg et al. (2014) evaluated the effectiveness of the Bali 2004 tsunami recovery interventions for assisting the community. A survey was conducted to assess health and housing outcomes.
For those evaluations that were mixed (i.e. included process and outcomes evaluations), the outcomes focused evaluation was often a small subsection of the reporting. For example, the final report by the Victorian Department of Sustainability and Environment (2010) on the 2007 Gippsland Flood/Storm Recovery Program focused predominantly on the processes of implementation. A small section of the report looked at the outcomes of environmental recovery efforts measured through fauna field surveys. Similarly, an evaluation of the Canberra 2003 bushfire by Camilleri et al. (2007) reviewed recovery programs that were implemented, including the communication strategies and community involvement, with a minor section of the report dedicated to assessing the mental health outcomes.
There were, however, some examples where outcomes were measured across the social, economic, built, and environmental dimensions of recovery. Notably, the Canterbury Earthquake Recovery Authority (2014) examined the outcomes of post-disaster recovery efforts for the 2011 Christchurch earthquake. The evaluation included indicators for economic recovery, social recovery, the built environment, and the natural environment (Table 2).
Data were also collected for the evaluation from a survey of residents (including process elements), and a review of secondary data. The outcomes indicators provide a useful case study for practitioners to consider how outcomes may be incorporated into an evaluation, and may provide indicators that could be incorporated into a national post-disaster recovery framework.
The Data Center, an independent organisation in Louisiana, USA, also provide a holistic assessment of post-disaster recovery that examines outcome indicators. Hurricane Katrina, that hit New Orleans in 2005, had a devastating impact on the community. Recovery has been an extended, slow process. The Data Center tracks recovery from the disaster against indicators measuring population, economy, housing, infrastructure, and environmental sustainability. Table 3 presents a summary of these indicators. Although there is more of an emphasis on social economic recovery, the Data Center evaluation is a useful case study of outcomes indicators, particularly over the long-term.
Evaluation type and data collection methods
The most common type of evaluation identified by the review was government review/inquiry (Table 5), for example the government inquiries into the 2009 flood on the mid and North Coast of New South Wales (Recovery Coordinator 2009) and the 2013 bushfire in Victoria (Inspector-General for Emergency Management 2016). These occurred between 6 months and 2 years following a disaster event. This type of evaluation focused on the implementation of programs, resource allocation, and governance around disaster management. For government inquiries such as the one examining the 2009 Victorian bushfires (Victorian Bushfire Royal Commission 2010) and the Queensland floods in 2010–2011, recovery was only included as a small part of the inquiry. Response and immediate relief were the focus of these inquiries, with emphasis on the organisation of response effort, the decision making process that guided response and relief, and communication.
The short timescale for government reviews/inquiries may explain the focus on response and immediate relief compared to the Hurricane Katrina and Christchurch earthquake evaluations (5–10 years and likely to be ongoing). The length of time for recovery may be highly variable, influenced by the type, scale and impact of the disaster event and the surrounding context (e.g. location, socio-economic characteristic of the community, industry etc.). It is unclear how long after a disaster event ‘recovery’ is expected to be achieved, however this is more likely to be decades, rather than months or even years. The lack of a clearly defined endpoint presents a challenge for practitioners evaluating post-disaster recovery interventions. The Hurricane Katrina and Christchurch earthquake examples suggest that an appropriate way to manage this uncertainty (particularly for large scale events) is to iteratively monitor recovery at regular intervals over a 10 year period.
The evaluations found through the grey literature review often relied on the experiences of those involved in the recovery effort to inform the evaluation (Table 5). This included personnel from government agencies, members of the recovery task force, and volunteers or those involved with non-government agencies. These personal accounts provided detailed information on what happened during the disaster management process. However, they provided limited information on the outcomes of the post-disaster recovery. Similarly, records and reports on the process were often reviewed, however these too only provide information on the disaster management processes.
For outcomes focused evaluation, quantitative indicators were generally used. This included GDP (Lucich et al. 2006), community attitudes revealed by surveys (Camilleri et al. 2007; Frankenberg et al. 2014; UNICEF 2009), revenue (Virginia Horticultural Centre South Australia 2008), employment figures (Regional Australia Institute 2013), school performance (Canterbury Earthquake Recovery Authority 2014) or ecological field surveys (Robichaud et al. 2010; Victorian Government Department of Sustainability and Environment 2005). These data were compared with conditions before the disaster event or to trends that occurred in comparable areas that were not affected by the disaster.
The location and context of the Hurricane Katrina and Christchurch earthquake disaster events may have helped authorities be able to develop and measure outcomes indicators. Both events were relatively contained in terms of geographic location and occurred in highly populated urban areas where data on population characteristics and records of infrastructure and environmental health are more likely to be collected as a matter of course. As such, a wider range of indicators may be available in these areas that allow for pre and post disaster comparison.
For disasters that have a more widespread impact and cross jurisdictional boundaries, or occur in more remote locations, evaluators may not have access to these types of data. In these circumstances, qualitative indicators informed by community and other stakeholders’ experience may be relied upon. This is a further indicator of how important it is to have the community at the center of post-disaster recovery evaluation. Fisher and Talve (2011) provide a useful example from an evaluation of the effectiveness of arts programs for community recovery from the 2009 Victorian bushfires. Interviews with internal and external stakeholders as well as five multimedia case studies were used to develop insights into the contribution of the arts programs in galvanising, uniting and healing the community. Similarly, the effectiveness of interventions to assist social and economic recovery from the 2005 Gawler River flood in South Australia were evaluated using interviews with the community and those involved in the recovery effort (Department for Families and Communities n.d.).
For practitioners looking to evaluate post-disaster recovery efforts, it is important to look beyond the experiences of those involved and operational records about what happened. To assess the success of post-disaster recovery efforts, it is important to consider the changes that have occurred. The examples identified through this review provide some useful tools to achieve this. However, there were no examples identified that made the critical link between program intervention, impact, and outcomes. The next section of this paper discusses some useful methods for outcomes focused evaluations.
This review demonstrates a clear need for more consistent and comprehensive evaluations of post-disaster recovery that link interventions to impact and outcomes. Currently, post-disaster recovery in Australia and New Zealand takes a significant amount of public expenditure. As discussed above, recovery from the Christchurch earthquake is predicted to take the equivalent of about 20 percent annual GDP for New Zealand, while the Australian Government continues to expend considerable resources on recovery from previous disaster events (Productivity Commission 2014; The Treasury 2013). Yet it is unclear whether this spend is justified and whether the resources are being efficiently allocated to deliver effective outcomes that support community recovery.
There are a number of reasons why evaluations are not conducted, including unclear definition of post-disaster recovery, the types of activities and actions that characterise this phase of disaster management, and what ‘success’ looks like. The New Zealand and Australian government adopt a holistic framework for post-disaster recovery that places community at the core of the recovery process. However, what this means in terms of outcomes for the community is challenging to define.
These challenges to evaluating post-disaster recovery, and lack of outcomes-focused evaluations identified from the review, provide strong justification for the development of a national monitoring and evaluation framework. The framework should provide practitioners with a clear understanding of what post-disaster recovery is and what success may look like. It should also provide guidance on timing and methodology. Critically, the framework needs to adopt a methodology to ensure practitioners can connect interventions with impacts and outcomes, and determine whether resources are being allocated efficiently to support community recovery.
There are a number of methodologies and approaches for program evaluation that may be useful. Theory of Change (TOC) is a method used for evaluations that maps out the short and mid-term outcomes that are expected to result in the achievement of the long-term goals of the interventions (Weiss 1995). The approach is most useful for complex, long term social issues and may be linked to logic models that explicitly connect the programs inputs and actions with the outputs and desired outcomes. TOC may be appropriate for the complex policy environment of post-disaster recovery. Connell and Kubisch (1998: 1) promote TOC as an appropriate methodology to evaluate comprehensive community initiatives with
multiple strands (economic, political, and social), which operate at many levels (community, institutional, personal network, family, and individual), are co-constructed in a collaborative process by diverse stakeholders, and evolve over the course of the initiative.
For post-disaster recovery practitioners, developing a TOC and associated logic models could be a part of the planning process for the intervention and may be a useful framework for evaluations.
Importantly, the results of an evaluation should be useful and able to drive changes that will improve the policy intervention. Goni (2012) highlight this issue in an analysis of evaluations for public expenditure management in OECD countries. Of the seven countries studied, Goni (2012) found that although most implemented public expenditure evaluations, there was little evidence of these being used to inform spending management behaviour. The author suggests that evaluations need to be tailored to the complexity of public management context to increase their usefulness for decision making.
For post-disaster recovery, it is imperative that government spending is efficient, with resources allocated to interventions that are effective for supporting community recovery. Evaluating spending efficiency is one approach that could be used to determine whether resources are being efficiently converted to outcomes, and to identify how resources could be better allocated. This may be integrated into a TOC. Brecher et al. (2005) demonstrate how an evaluation can be used to test the TOC and help improve the delivery of a program or intervention. The authors conducted a fiscal analysis evaluation during the interim period of a health initiative and use the findings to redefine the goals of resource allocation and modify the TOC to better achieve the desired outcomes. Seifert and Nieswand (2014) demonstrate how spending efficiency evaluation techniques can identify inefficiencies and areas of improvement in local government spending. Efficiency analysis is used as a benchmarking approach to compare the transformation of one unit of resource input into output. Using this approach, unit-level inefficiencies in government departments in metropolitan France, and the factors that led to these inefficiencies, were identified. A similar approach was used by Afonso and Fernandez (2007) to measure local government spending in Lisbon. The results of their spending efficiency evaluation indicate that the same level of output could be achieved using one third less resources.
In a post-disaster recovery evaluation, incorporating spend efficiency using a spend efficiency tool may allow practitioners to identify which interventions are delivering most benefit for community recovery relative to input. The benefit to community may be determined as meeting their needs (commonly identified in a needs assessment immediately following a disaster) and progress towards recovery. In addition, adopting spend efficiency analysis in a national monitoring and evaluation framework may allow for comparison between post-disaster recovery interventions from comparable disaster events. This could be highly beneficial for identifying ‘best practice’ interventions that meet community needs and support community recovery. However, given the complexity of post-disaster recovery, measuring spend efficiency may present a significant challenge. Further research on the merits of spend analysis tools is required.
An important element of an evaluation is understanding how decisions are made about the allocation of resources. Ideally, resources are allocated efficiently to address the actual needs that drove the intervention, and deliver the desired outcomes. In reality, however, there are numerous factors that influence resource allocation that may or may not reflect the actual needs. Hajnal and Trounstine (2010) examine what influences local decision making by analysing government spending patterns from a range of nationally representative studies. They find that economic constraints are critical influences in decision making on resource allocation. The authors further note that, in addition to actual needs, ‘redistributional, allocational, and developmental spending is also strongly influenced by political imperatives [and] institutional constraints’ (Hajnal and Trounstine 2010, 1130). For post-disaster recovery, there are likely to be similar influences on how and where resources are allocated that are independent of the actual needs of the community. Without evaluations being consistently conducted, however, it is unclear what these influences are and whether they are impeding effective post-disaster recovery.
Post-disaster recovery: A complex policy environment
Recovery from a disaster event is a complex process. The disasters reviewed in this paper included multiple levels of government, non-government organisations, and volunteers, and covered social, economic, built, and environment domains. Community involvement in the recovery process was identified in the literature as a key to success. However, this too adds to the complexity of the recovery process and attempts at evaluating it. There is a need to conduct these evaluations in a consistent manner to learn from experience and be able to target resources in disaster recovery.
A national framework for disaster recovery is an important step in achieving this. This will provide a useful starting place in knowing where to begin in this complex policy and data collection environment. It will also help achieve consistency in the evaluations undertaken and help lift post-disaster recovery evaluations from being a small addition to response and relief reviews to being a meaningful evaluation of outcomes. It may also help ensure that evaluations are conducted across the whole recovery effort, rather than focusing on specific sub elements or the disaster management process. The theory of change method is a useful approach for developing the overarching framework, as it is appropriate for the complex, long-term, multi-stakeholder nature of post-disaster recovery.
To develop a theory of change, measures of the activities and outcomes of the intervention are required. This typology and review may be used to identify disaster events that provided useful methods to measure activities and outcomes, and thus assist with building a theory of change and the development of a national framework. Useful examples include disaster events where a post-disaster recovery evaluation has been undertaken; the evaluation is outcomes focused (with sufficient attention to process); and it looks at the social, economic, built, and environmental dimensions of recovery. The holistic, outcomes focused evaluations by the Canterbury Earthquake Recovery Authority (2014) and Plyer et al. (2013) are some examples that may be useful for further investigation.
In addition, there is merit in further investigation of how spending analysis evaluation tools could be applied to the post-disaster recovery context. Given the significant amount of public expenditure on post-disaster recovery, efficiency or fiscal analysis would help identify areas of improvement that could increase the efficiency of the public expenditure. Government intervention for post-disaster recovery is critical for helping communities following a disaster event, and knowledge built through evaluation practice ensures resources are used in a manner that best achieves this aim.
The following points summarise the implications of this review for policy and practice:
Post-disaster recovery evaluation, conducted within an agreed monitoring and reporting framework, should be conducted for disaster events. Currently, there is a greater emphasis on reviewing the immediate response and relief efforts.
Although process evaluations are useful and important, they do not illuminate the full impact or outcomes of the post-disaster recovery effort. Outcomes focused evaluations can assess the effectiveness, efficiency and appropriateness of post-disaster interventions.
Theory of change methodology can be used to develop evaluation approaches suited to the complex policy and intervention environment of post-disaster recovery.
Outcomes based evaluations identified such as those deployed by the Canterbury Earthquake Recovery Authority (2014) and Plyer et al. (2013) may prove useful examples to identify measures of outcomes to inform a theory of change for post-disaster recovery evaluation.