Evaluations of post-disaster recovery: A review of practice material

Publications

Share / Export Citation / Email / Print / Text size:

Evidence Base

Australia and New Zealand School of Government

Subject: Business & Management

GET ALERTS

eISSN: 1838-9422

DESCRIPTION

202
Reader(s)
236
Visit(s)
0
Comment(s)
0
Share(s)

SEARCH WITHIN CONTENT

FIND ARTICLE

Volume / Issue / page

Related articles

VOLUME 2016 , ISSUE 4 (January 2016) > List of articles

  • |

Evaluations of post-disaster recovery: A review of practice material

Roberta Ryan * / Liana Wortley / Éidín Ní Shé

Citation Information : Evidence Base. VOLUME 2016 , ISSUE 4 , Pages 1-33 , ISSN (Online) 1838-9422, DOI: 10.21307/eb-2016-001, January 2016 © 2016.© The Australia and New Zealand School of Government

License : (CC-BY-NC-ND 4.0)

Published Online: 2016

ARTICLE

ABSTRACT

This paper reviews evaluations of post-disaster recovery efforts. The focus is on operational material and other ‘grey literature’ from disasters that have occurred in Australia, New Zealand and internationally. We develop a typology that categorises disaster events and includes whether evaluations were undertaken; the methods used; and whether the evaluations focused on the processes or outcomes of the recovery program. The review finds a lack of evaluation of post-disaster recovery. Where evaluations have been conducted, they are mostly process- rather than outcomes-based. There is a need for guidance for post-disaster recovery programs to support evaluation practice to determine the effectiveness, efficiency and appropriateness of post-disaster recovery interventions. There is significant investment in post-disaster recovery programs, with little known of their effectiveness. This review identifies useful case studies and methods to evaluate post-disaster recovery efforts, and informs the development of a national post-disaster evaluation framework.

Graphical ABSTRACT

Disaster events are a ‘condition or situation of significant destruction, disruption, and/or distress to the community’ (Commonwealth of Australia 1998, ix). Such events can have severe, long term social, economic and environmental impacts. In New Zealand, the 2010 and 2011 Canterbury earthquakes were estimated to have a financial cost of approximately $15 billion, and reduced total Gross Domestic Product (GDP) in 2011 by around 1.5% (NZ Treasury 2011, 96). In Australia since 2009, natural disasters have claimed more than 200 lives, destroyed 2670 houses and damaged a further 7680, and affected the lives and livelihoods of hundreds of thousands of Australians (Productivity Commission 2014, 3). Human-caused disasters, such as acts of terrorism, nuclear accidents, anthropogenic fires, and transport incidents, have similarly devastating impacts. For example, the World Health Organisation’s (WHO) extensive study of the health impacts of the 1986 Chernobyl nuclear disaster concluded that the accident was responsible for long-term physical health problems and deaths from radiation exposure, and had significant impacts on mental health and wellbeing of the general population (Bennett et al. 2006, 69-96).

Disaster or emergency management aims to reduce the short and long-term impact of a disaster event. It includes pre-disaster interventions to reduce the potential future impact and help community preparedness, immediate response and relief efforts, and ‘post-disaster recovery’ that commonly refers to the period of time and activities that occur after the immediate relief and response to a disaster event (Commonwealth of Australia 2011, 29; FEMA 2011, 8; Ministry of Civil Defence & Management n.d.). Effective post-disaster recovery is critical to getting community members ‘back on their feet’. However, Archer et al. (2015) concluded, based on a review of post-disaster literature, that this phase of disaster management is poorly defined. This paper will focus on post-disaster recovery as it is a critical phase of the disaster management process in need of greater clarity.

Post-disaster recovery is highly complex. It occurs in an environment of high stress, involves multiple agencies and stakeholders, has multiple priorities that evolve over time, and has no clearly demarcated end point (Commonwealth of Australia 2011, 3–6). Government intervention is generally required to assist the affected community during recovery. In Australia and New Zealand, each tier of government provides assistance to support recovery.

In New Zealand, the legislative framework for emergency management is provided by the Civil Defence and Emergency Management ACT 2002 (CDEM). The Ministry for Civil Defence and Emergency Management is mandated to provide overarching guidance for post-disaster recovery. The framework, presented in Focus on Recovery: A Holistic Framework for Recovery in New Zealand, outlines the roles of different stakeholder groups. Consistent with the approach adopted in Australia, the community is at the center of recovery, with government and other stakeholder groups providing support and assistance. CDEM Groups – comprising local authorities working in partnership with emergency services and major utilities – have a coordinating role, and lead the development of recovery plans. These are enacted by government departments (with particular focus on local authorities) and non-government organisations (Ministry for Civil Defence & Emergency Management 2005, 3, 18–19)

In Australia, the federal government provides funding and support to state and local governments as well as to businesses and the community (Productivity Commission 2014, 8). State and territory government agencies are directly involved in the on-ground response and recovery effort, and include a disaster management group that provides overarching coordination and direction during and after a disaster event. Local governments play a key role throughout the disaster recovery process, and support the community over the long-term. In addition to the three tiers of government, there may also be a regional or district disaster management group that plays a coordinating role (Commonwealth of Australia 2011, 4–5).

Public expenditure on post-disaster recovery is significant. Over the past decade, the Australian Government alone has spent around $8 billion on post-disaster relief and recovery (Productivity Commission 2014, 3) and forward estimates indicate another $5.7 billion is expected to be spent on past disaster events (Australian Audit Office 2015, 132). The New Zealand government is estimated to have contributed $15.2 billion to recovery from the Christchurch earthquake, and forward estimates predict a total of $40 billion will be spent on the rebuilding effort (The Treasury 2013).

Given the significant amount of public expenditure and the importance of the post-disaster recovery phase, evaluation can be useful to ensure that resources are efficiently allocated to achieve effective outcomes. Evaluations are critical to facilitate learning and continued improvements to the post-disaster recovery process to achieve desired outcomes. A recent report on Evaluation in government by the UK National Audit Office identifies a key purpose of ex-post evaluations (i.e. evaluations undertaken after policy implementation) as ‘a means to improve existing policies and to better design future policies’ (National Audit Office 2013, 5). Further, as Brecher et al. (2005) note in their exploration of expenditure analysis as an evaluation tool, evaluators of long-term, complex interventions (of which post-disaster recovery is undeniably one) are often expected to provide interim feedback to determine whether a project is ‘on-track’. The findings from these interim evaluations are used to guide decision making on whether revisions are required for the project, or if indeed it should be continued at all.

Undertaking evaluation can cover a variety of questions and stages of a project. While many definitions of evaluation are used, the term generally encompasses the systematic collection and analysis of information to make judgments, usually about the effectiveness, efficiency and/or appropriateness of an activity (The Sphere Project 2015; Australasian Evaluation Society 2010; Owen 2006; Ryan 2014). Effectiveness refers to the ability of the program or activity to achieve the desired goals (i.e. did it work?), efficiency considers whether resources are being used wisely (i.e. the relationship between inputs and outputs), and appropriateness examines whether the program or intervention is suitable for meeting its objectives in the policy context (i.e. was it the right intervention for the need or stated problem?). Figure 1 outlines the types of questions that relate to the key evaluation themes of process, appropriateness, efficiency, and effectiveness.

Figure 1

The logic of a program and the relationship with key evaluation themes

Source: Ryan (2014)

10.4225_EB-2016-4_f001.jpg

Despite the potential value of evaluation, there is currently no existing national framework for monitoring or evaluating post-disaster recovery in either New Zealand or Australia. This is a significant barrier to jurisdictional learning from previous post-disaster experience, which would help to improve the effectiveness and efficiency of government interventions (at all levels) as well as provide rationale for program design and government investments. A recent review of Australian emergency management evaluations by Dufty (2013) demonstrated the current inconsistencies in post-disaster recovery evaluation. This brief review of a sample of evaluations found that often no evaluation was conducted at all. If they were conducted, the type and timing of the evaluation were highly variable. Similarly, the review by Archer et al. (2015) found that there was no common understanding, definition or indicators of success for post-disaster recovery. There has yet to be a systematic review of the post-disaster evaluations in New Zealand. However, without a national monitoring and evaluation framework, we anticipate there would be similar variations in the conduct of evaluations, as well as their type and timing. With the development of a framework outlining definitions, indicators of success and useful evaluation measures, evaluation practice would improve and increased learnings about effective interventions for post-disaster recovery could be fostered. Consistency in monitoring and evaluation of post-disaster recovery interventions would provide further guidance for practitioners in the field, and the basis on which to build an ongoing knowledge base.

This paper reviews existing evaluations of post-disaster recovery. It extends the initial review undertaken by Dufty (2013) and conducts a more extensive, systematic search and analysis of existing post-disaster evaluations from the Australian, New Zealand and international grey literature. The purpose of the review is to gain further insights into the extent and type of evaluations of post-disaster recovery interventions and the methods used in practice. To achieve this purpose, the type of interventions included in post-disaster recovery is investigated. This will contribute to understanding of the post-disaster recovery phase, and help to clarify it.

The review will identify trends in the methods used for evaluations, and aims to assist program managers to identify appropriate and comparable case studies to help guide evaluation of post-disaster recovery efforts. The findings of the review will be useful for informing the development of a post-disaster evaluation framework that will improve the application and use of evaluation. As disasters potentially impact all locations, and recovery commonly involves each level of government, the outcomes of this review are relevant to disaster recovery program managers across New Zealand, Australia, and internationally.

The search method

To identify evaluations of post-disaster recovery interventions, the literature search focused on operational material and other grey literature. We used this approach in order to distinguish evaluations conducted for academic purposes from empirical or practical evaluations that were done to assess the performance of a program or initiative. We conducted a search of the grey literature from national and international disasters identified from the Australian Emergency Management Knowledge Hub1 (the Hub). The Hub provides details of 680 disasters from Australia, New Zealand, and internationally. Dating back to 1791, it includes materials from natural and human caused disasters such as bushfires, cyclones, floods, shipwrecks, criminal activities (including acts of terrorism), and epidemics. To narrow the search the response was limited to disasters that occurred since 1995. We made the assumption that grey literature may be difficult to access online for events that occurred before this date. The focus here is on evaluations of post-disaster recovery for disasters that occurred before January 2015. Post-disaster recovery is an extended process that can take a number of years, and evaluations are thus unlikely to occur immediately after the event (Labadie 2008). This reduced the result to 217 disaster events. The database includes all disaster types. For this review, the focus was on:

  • Floods/storm surges/tsunami

  • Hurricane/cyclones

  • Earthquake

  • Bushfire

  • Criminal acts/terrorism

The choice of these disaster types reflects the all hazards approach to disaster management that is being adopted in Australia, New Zealand, and internationally. The all hazards approach to disaster management refers to the ability for arrangements and programs to deal with a wide variety of hazards, including natural and human caused disasters (Commonwealth of Australia 2011, 147). The disaster types that are the focus of this review included those that were most common according to the data provided on the Hub.

Using the above criteria, we identified 73 disasters. The Hub is the only catalogue of disasters for Australia and New Zealand, and it includes international examples. However, there were some notable disaster events that were not identified in the search. In particular, there was under-representation of disaster events outside Australia. To augment the findings from the Hub, we added an extra 11 prominent disaster events to the list. Prominent disasters were identified as those that received significant media attention, and were chosen through a Google search. The resulting list of disasters that were the focus of the grey literature search was 84.

To search the grey literature for existing evaluations for the identified disaster events, we searched the resource section of the Hub and the relevant government websites, and also conducted a general web search. The search terms included the name, location, type and date of the disaster as well as the terms recovery and evaluation/review/outcomes/success. We recorded and analysed the evaluations identified from the search using the typology presented in the following section.

A typology to categorise disaster events and evaluations

We developed a typology to review and analyse the grey literature identified through the search. The typology provides a useful framework for identifying case studies of evaluations. It can be used to inform the development of a post-disaster recovery evaluation framework and to assist in its practical application. A case study is a research strategy (Yin 1981) that enables ‘detailed contextual analysis of a limited number of events or conditions and their relationships’ (Dooley 2002, 335). It focuses on just one or a few instances of the phenomena being researched, and allows for an in-depth study. The case study draws on a range of methods of data collection, and uses triangulation of the evidence to investigate theoretical or stakeholder propositions (Yin 2003).

A draft typology was created prior to reviewing the identified grey literature. This included the basic elements to categorise a disaster (location, disaster type, date of occurrence and estimated impact), whether or not an evaluation was undertaken, and the evaluation methods. The typology was revised and amended as additional elements emerged that were informative for categorising the disaster events, may potentially impact on evaluation, and assisted in the analysis and categorising of the evaluation methods. Table 1 presents the typology and description of each category.

Table 1

A typology for post-disaster recovery evaluations

Category

Description

Disaster characteristics

Disaster type, location and date of occurrence

The characteristics of the disaster were recorded including type (e.g. cyclone, bushfire etc), location and date of occurrence. This information identifies the individual events and any trends between these characteristics and the evaluations conducted.

Indicators of extent of impact

The impact of the disaster affects the recovery effort. The estimated impact was difficult to identify from the grey literature as there are a number of different ways in which this can be assessed. The scope was narrowed down to the immediate social impact represented by loss of life, and the estimated economic impact as the insured cost. The actual impact will be much more complex, however, further investigating this is beyond the scope of this review. The insured cost for the disaster events was provided on the Hub and normalised for current cost estimates. For additional disasters not listed on the Hub, the insured cost was reported where it could be found from the grey literature.

Post-disaster recovery effort

Types of activities/programs undertaken

The activities and programs for post-disaster recovery are extensive. The Australian Government’s Community Recovery Handbook 2 promotes a holistic approach to post-disaster recovery that covers social, economic, built and environmental dimensions.

Community involvement in post-disaster recovery planning

There is general agreement in the literature that community involvement in the post-disaster recovery planning process is key to achieving good outcomes, and it is one of the principles of post-disaster recovery outlined by the Australian Government (Commonwealth of Australia 2011).

Agencies involved in recovery

The policy context for post-disaster recovery is complex and frequently involves multiple agencies. In Australia, this includes local governments, state or territory government departments, the federal government, and NGOs and volunteer organisations. In New Zealand, local and regional authorities commonly take the lead to support communities and are assisted by the federal government, NGOs and volunteer organisations Internationally, post-disaster recovery similarly involves a multi-level governmental response. International organisations and support from other countries may also occur.

Post-disaster recovery evaluation

Number of evaluations conducted (if any)

The number of evaluation documents identified from the literature search.

Evaluation type

As identified by Dufty (2013), there are different types of post-disaster recovery evaluations. These include government inquiries, independent evaluations and operational reviews.

Who undertook the evaluation

The party that undertook the evaluation was recorded. This may include a government department, external consultancy, independent researcher or NGO.

Process or outcomes focused evaluation

Evaluations are commonly process focused or outcomes focused.

Process evaluations look at program implementation or resource allocation, whereas outcomes focused evaluations assess the impact of the activity or intervention.

Data collection methods

The method for data collection to inform the evaluation.

Focus of the evaluation

The aspects of post-disaster recovery that the evaluation focused on.

This was further categorised into social, economic, built or environmental domains.

Publication details

Bibliographic details of the evaluation.

The publication details of the evaluation document.

To present the data, the typology was separated into two tables2 (Appendix 1). Table 4 presents the disaster characteristics and the evaluations that were identified by the grey literature search. Table 5 provides further details on the evaluations conducted and the information source. Of the 84 disasters identified for the review, post-disaster recovery evaluations were only found for 35 disasters. For the sake of brevity, we include in the tables only those disasters where an evaluation of post-disaster recovery was identified.

Findings

Disaster characteristics and post-disaster recovery actions

Disaster events included in the review (summarised in Tables 2 and 3) occurred in Australia, New Zealand, USA, Canada, UK, Switzerland, Japan, Indonesia and the Pacific Islands. The nature of the search resulted in a strong emphasis on Australian disaster events as the Hub is an Australian based catalogue of disaster events. Bushfires and floods were the most common disaster type identified, with only eight criminal acts/terrorist events, three earthquakes, and two tsunamis reviewed. The scale of the impacts ranged from relatively minor (for example the bushfire in Coonabarabran, NSW in 2013) to devastatingly high (for example Hurricane Katrina in New Orleans). However, it is worth noting again that these are only crude indicators of the impacts of a disaster. The actual impact will be much more complex, and investigating this in further detail is beyond the scope of this review.

Table 2

Outcome indicators for post-disaster recovery

Recovery components

Headline indicators

Economic

  • Business activity

  • Economic output

  • Economic confidence

  • Labour market

  • Central city activity

Social recovery

  • Quality of life

  • Educational achievement

  • Mental wellbeing

  • Social connectedness

  • Offending patterns

  • Housing affordability

Built environment

  • Land supply

  • Central city repair and rebuild

  • Horizontal infrastructure repair

  • Ease of travel and transportation

Natural environment

  • Air quality

  • Biodiversity

  • Drinking water sources

  • Waterway health

Table 3

Outcome indicators for post-disaster recovery from Plyer et al. (2013)

Recovery components

Indicators

Economic Growth

Job growth

Drivers of the economy

Local-serving clusters

Wages

Productivity

Airport traffic

Entrepreneurship

Venture capital

Educated workforce

State funding for higher education

Job sprawl

Inclusion

Median household income by race and ethnicity

Educational attainment by race/ethnicity and sex

Jail incarceration rates

Size of city’s middle class

Size of city’s middle class by race and ethnicity

Income inequality

Suburbanization of poverty

Quality of life

Arts and culture

Public education

High school cohort graduation rates

Youth investment

Public safety

Public corruption

Housing affordability

Sustainability

Bike pathways

Commuting by public transit

Air quality

Groundwater salinity

Coastal wetlands

Most post-disaster recovery efforts addressed the social, built, economic and environmental domains. The types of recovery activities included:

  • Social: physical and psychosocial support such as healthcare, counselling and programs targeted to increasing community welfare such as art initiatives or memorials.

  • Economic: support to buffer and improve the local economy. This may include stimulus activities, assistance to primary industries or tourism, employment programs, or business counselling development.

  • Built: rebuilding physical infrastructure including housing, roads, bridges and other development. May also include re-zoning or relocation of residents.

  • Environment: restoring environments affected by the disaster. May include revegetation, monitoring and clearing waterways, stabilising coastal zones, or other activities to assist impacted ecosystems.

Government agencies and practitioners are adopting a holistic approach to recovery. In Australia and New Zealand, disaster recovery frameworks have been established to guide the development of post-disaster recovery programs. In Australia, the Community Recovery Handbook 2, released by the Australian Government Attorney General’s Department, states that community recovery should be coordinated across social, built, economic, and environment domains (Commonwealth of Australia 2011, 78). In New Zealand, the framework for recovery established by the New Zealand Ministry of Civil Defence and Emergency Management (2005, 6) similarly presents recovery as an integration of social, built, economic, and environment domains to support the community.

For some disasters, particularly acts of terrorism or criminal activity, recovery efforts focused on social and built domains. For example, recovery efforts for the Oklahoma City bombing in 1995 focused on rebuilding the affected area and providing services to address physical and mental health (Oklahoma Department of Civil Emergency Management n.d.). Similarly, recovery activities following the 9/11 terrorist attacks focused on social, built and economic aspects (US Government 2006). In these examples, not addressing environmental dimensions was appropriate. This highlights a potential issue with the all hazards approach to disaster management. Although a holistic approach for disaster recovery could be appropriate for most disaster events, it may not be appropriate for all. Similarly a national framework for post-disaster recovery evaluation will need to take into account the variability of disaster events and post-disaster recovery actions. While overarching frameworks and guidelines are useful for guiding practice, they should permit tailoring of activities or evaluations to the specific context.

The involvement of the community was variable in post-disaster recovery. Some post-disaster recovery efforts involved significant community involvement. For example, the community was heavily involved in planning disaster recovery following the Blue Mountains bushfire in 2013. Community meetings as well as online forums and surveys were used to identify community priorities (NSW Government 2013). For other disaster events, there appeared to be less focus on the community. This does not necessarily mean that community consultation did not occur. It may be that this aspect of the disaster recovery was simply not well documented. This would be expected as documentation in general for some disaster events was low.

Nonetheless, community involvement should be a high priority for post-disaster recovery. The importance of a community focused approach has been recognised in the disaster recovery frameworks for New Zealand and Australia, where community lies at the core of recovery, and the recovery principles state that the process should be ‘community led’ (Commonwealth of Australia 2011, 78; Ministry of Civil Defence & Emergency Management 2005, 6). The limited inclusion of community perspectives in some cases suggests that a gap exists between the theory of post-disaster recovery promoted at a national level and what is occurring on the ground.

Leadbeater (2013) argues strongly for the necessity of disaster recovery being community focused and tailored to community needs. In particular she highlights the importance of integrating appropriate community leaders into the planning process so that it is tailored to the existing values, networks, projects, relationships, knowledge, and capacity of the community. Similarly, Hawkins and Maurer (2010) argue that existing social capital in the community was key in assisting families to recover following Hurricane Katrina. The disaster events identified by the review that demonstrated high levels of community involvement may provide useful examples or guides for practitioners developing a community engagement strategy for disaster recovery.

A need for more consistent evaluations of post-disaster recovery

Consistent with the findings of Dufty (2013), our more extensive grey literature review demonstrates that evaluations of post-disaster recovery are often not undertaken in practice. Of the 84 disasters identified in the review, evaluations were identified for only 35 disaster events. The remaining 49 disaster events either did not have an evaluation, or it was not published or made publicly available. Given that it is common practice for government agencies to release these reports, we predict that the former is much more likely the case.

The lack of evaluations is a significant concern for the advancement of post-disaster recovery efforts and ensuring resources are efficiently allocated to achieve good outcomes. An evaluation determines how well a government program or intervention has met its objectives, holds officials accountable for its implementation, and provides insights for future policy making, including whether resources should be continued, increased or reduced (Althaus et al. 2007, 179).

National guidelines and frameworks for post-disaster recovery emphasise that community should be the focus for programs, with interventions intended to support communities in their recovery. However, the lack of evaluations means we do not know whether the community is actually benefiting from interventions. Without examining the appropriateness, effectiveness and efficiency of interventions, we cannot determine whether the programs being implemented are contributing to recovery and delivering outcomes. As such, it is unknown if the current spending on post-disaster recovery is justified and the most efficient use of resources, or whether funding and intervention approaches require a re-think.

Insufficient monitoring and evaluation is not uncommon for government interventions. The UK National Audit Office and the Public Accounts Committee (2013, 6) have criticised government evaluations due to:

  • ‘gaps in the coverage of evaluation evidence;

  • poor-quality evaluation;

  • insufficient use of evaluation evidence; and

  • difficulties faced by independent researchers in accessing administrative data and other government data to conduct their own evaluations of government interventions.’

There are a number of reasons why evaluations may be limited. Insufficient resourcing is a commonly identified barrier to evaluating a government program or intervention (DeLuca et al. 2010). Within a limited resource environment, priority is given to the implementation of the program. Unclear definitions and confusion over what is being evaluated also present a significant barrier. In recent discussions with disaster recovery personnel across Australia, it became apparent that there is a lack of understanding about:

  1. what defines ‘post-disaster recovery’;

  2. the steps or interventions that are included in this phase of disaster management, and;

  3. what ‘success’ may look like.

This is further complicated by no clear end-point for disaster recovery intervention. Without these aspects being clearly defined, it is extremely challenging for practitioners to design and undertake evaluations. A national monitoring and evaluation framework should go some way to addressing these issues.

Disaster events identified by the review for which an evaluation was undertaken were often more recent, had a large social or economic impact, or were in more populated areas. The finding that evaluations were more common for recent events may reflect an increased emphasis on government transparency in the last 10–15 years. Governments are increasingly required to be more accountable for the resources being spent on implementing programs. As noted by Bovens (2007, 182), accountability of administrative agencies is the ‘hallmark of modern democratic governance’. Shkabatur (2012, 82) defines accountability as consisting of two elements: ‘the explanation and justification of agencies’ activities to the public; and an accompanying mechanism for public sanctions’. Evaluations address the first of these elements and, as outlined by Chouinard (2013, 238), are often defined as a ‘neutral instrument providing impartial, evidence-based, and objective information intended primarily to satisfy accountability requirements’. The notion that evaluations are becoming more frequent due to increased pressure on governments to be accountable for public spending is consistent with the association between the scale of impact and presence of evaluations. Those that have required greater resource investment may be more held to account for this resource investment.

Process versus outcomes evaluations

The review of academic disaster literature by Archer et al. (2015) found a strong emphasis on evaluating the effectiveness of the recovery process, as opposed to the impact and outcomes of the process. The following provides an example of the difference between outcomes and process focused evaluations.

  • A process evaluation looks at the actual development and implementation of a particular program. It establishes whether a numeric target has been reached and strategies implemented as planned. For example a process evaluation might confirm that 400 people applied for housing support after a flooding and that 350 were granted the support.

  • An outcomes focused evaluation is a systematic approach that measures the impacts, benefits, or changes that have occurred as a result of a particular program. For example a process evaluation can provide the quantifiable number of job reskilling that occurred following the earthquake devastation of a rural town’s only factory. The outcomes approach would tell you how many of those demonstrated increased confidence, changed behaviours, found jobs because of the new skills, etc.

The few existing evaluations identified by the review were mostly process evaluations (Table 4). Process evaluations seek to involve stakeholders in considering how activities occurred. The evaluation method seeks their views usually through focus groups, interviews or workshops. Examples include the after-action review undertaken for the bushfire in Coonabaraban, NSW in 2013 (Warrumbungle Shire Council 2013) and the government review of post-disaster recovery from the 2013 Tasmanian bushfires (Tasmanian Bushfire Inquiry 2013).

Evaluations that considered outcomes were generally undertaken by independent researchers or external consultancies. The outcomes evaluations identified by this review typically focused on the effectiveness of an intervention in enabling change by measuring a specific variable. This approach provides a data snap-shot, which while potentially useful for examining change over time, fails to connect program interventions with impact and outcomes. Recovery processes, outcomes and impact are inextricably linked. To determine whether specific programs are actually helping the community to recover, outcomes evaluations need to demonstrate the causal relationship between the intervention (and government spend), impact, and outcome. A sole focus on indicators without linking to the program or interventions is a barrier to researchers and evaluators making in-depth insights, with any judgments of success limited to the entire recovery program, or entire domains.

The snap-shot evaluations identified from our literature review represent an ‘outcomes only’ evaluation rather than ‘outcomes focused’ evaluation. They provide examples of useful indicators that may help inform outcomes focused evaluation, but further work is required to establish a causal link with interventions and impacts.

Evaluations that did consider outcomes tended to only focus on a specific aspect of post-disaster recovery – in particular, social and economic outcomes measured as mental health or wellbeing and economic activity (e.g. GDP, tourism income, industry profit). For example, the Mount Sinai Medical Center (2011) conducted long-term clinical assessments of physical and mental health to evaluate the effectiveness of support programs for those affected by the 9/11 terrorism event. The Regional Australia Institute (2013) examined the effectiveness of government recovery programs following Cyclone Yasi in 2011 using indicators of population growth, return of the tourism trade, performance of the agricultural sector, and employment levels. Frankenberg et al. (2014) evaluated the effectiveness of the Bali 2004 tsunami recovery interventions for assisting the community. A survey was conducted to assess health and housing outcomes.

For those evaluations that were mixed (i.e. included process and outcomes evaluations), the outcomes focused evaluation was often a small subsection of the reporting. For example, the final report by the Victorian Department of Sustainability and Environment (2010) on the 2007 Gippsland Flood/Storm Recovery Program focused predominantly on the processes of implementation. A small section of the report looked at the outcomes of environmental recovery efforts measured through fauna field surveys. Similarly, an evaluation of the Canberra 2003 bushfire by Camilleri et al. (2007) reviewed recovery programs that were implemented, including the communication strategies and community involvement, with a minor section of the report dedicated to assessing the mental health outcomes.

There were, however, some examples where outcomes were measured across the social, economic, built, and environmental dimensions of recovery. Notably, the Canterbury Earthquake Recovery Authority (2014) examined the outcomes of post-disaster recovery efforts for the 2011 Christchurch earthquake. The evaluation included indicators for economic recovery, social recovery, the built environment, and the natural environment (Table 2).

Data were also collected for the evaluation from a survey of residents (including process elements), and a review of secondary data. The outcomes indicators provide a useful case study for practitioners to consider how outcomes may be incorporated into an evaluation, and may provide indicators that could be incorporated into a national post-disaster recovery framework.

The Data Center, an independent organisation in Louisiana, USA, also provide a holistic assessment of post-disaster recovery that examines outcome indicators. Hurricane Katrina, that hit New Orleans in 2005, had a devastating impact on the community. Recovery has been an extended, slow process. The Data Center tracks recovery from the disaster against indicators measuring population, economy, housing, infrastructure, and environmental sustainability. Table 3 presents a summary of these indicators. Although there is more of an emphasis on social economic recovery, the Data Center evaluation is a useful case study of outcomes indicators, particularly over the long-term.

Evaluation type and data collection methods

The most common type of evaluation identified by the review was government review/inquiry (Table 5), for example the government inquiries into the 2009 flood on the mid and North Coast of New South Wales (Recovery Coordinator 2009) and the 2013 bushfire in Victoria (Inspector-General for Emergency Management 2016). These occurred between 6 months and 2 years following a disaster event. This type of evaluation focused on the implementation of programs, resource allocation, and governance around disaster management. For government inquiries such as the one examining the 2009 Victorian bushfires (Victorian Bushfire Royal Commission 2010) and the Queensland floods in 2010–2011, recovery was only included as a small part of the inquiry. Response and immediate relief were the focus of these inquiries, with emphasis on the organisation of response effort, the decision making process that guided response and relief, and communication.

The short timescale for government reviews/inquiries may explain the focus on response and immediate relief compared to the Hurricane Katrina and Christchurch earthquake evaluations (5–10 years and likely to be ongoing). The length of time for recovery may be highly variable, influenced by the type, scale and impact of the disaster event and the surrounding context (e.g. location, socio-economic characteristic of the community, industry etc.). It is unclear how long after a disaster event ‘recovery’ is expected to be achieved, however this is more likely to be decades, rather than months or even years. The lack of a clearly defined endpoint presents a challenge for practitioners evaluating post-disaster recovery interventions. The Hurricane Katrina and Christchurch earthquake examples suggest that an appropriate way to manage this uncertainty (particularly for large scale events) is to iteratively monitor recovery at regular intervals over a 10 year period.

The evaluations found through the grey literature review often relied on the experiences of those involved in the recovery effort to inform the evaluation (Table 5). This included personnel from government agencies, members of the recovery task force, and volunteers or those involved with non-government agencies. These personal accounts provided detailed information on what happened during the disaster management process. However, they provided limited information on the outcomes of the post-disaster recovery. Similarly, records and reports on the process were often reviewed, however these too only provide information on the disaster management processes.

For outcomes focused evaluation, quantitative indicators were generally used. This included GDP (Lucich et al. 2006), community attitudes revealed by surveys (Camilleri et al. 2007; Frankenberg et al. 2014; UNICEF 2009), revenue (Virginia Horticultural Centre South Australia 2008), employment figures (Regional Australia Institute 2013), school performance (Canterbury Earthquake Recovery Authority 2014) or ecological field surveys (Robichaud et al. 2010; Victorian Government Department of Sustainability and Environment 2005). These data were compared with conditions before the disaster event or to trends that occurred in comparable areas that were not affected by the disaster.

The location and context of the Hurricane Katrina and Christchurch earthquake disaster events may have helped authorities be able to develop and measure outcomes indicators. Both events were relatively contained in terms of geographic location and occurred in highly populated urban areas where data on population characteristics and records of infrastructure and environmental health are more likely to be collected as a matter of course. As such, a wider range of indicators may be available in these areas that allow for pre and post disaster comparison.

For disasters that have a more widespread impact and cross jurisdictional boundaries, or occur in more remote locations, evaluators may not have access to these types of data. In these circumstances, qualitative indicators informed by community and other stakeholders’ experience may be relied upon. This is a further indicator of how important it is to have the community at the center of post-disaster recovery evaluation. Fisher and Talve (2011) provide a useful example from an evaluation of the effectiveness of arts programs for community recovery from the 2009 Victorian bushfires. Interviews with internal and external stakeholders as well as five multimedia case studies were used to develop insights into the contribution of the arts programs in galvanising, uniting and healing the community. Similarly, the effectiveness of interventions to assist social and economic recovery from the 2005 Gawler River flood in South Australia were evaluated using interviews with the community and those involved in the recovery effort (Department for Families and Communities n.d.).

For practitioners looking to evaluate post-disaster recovery efforts, it is important to look beyond the experiences of those involved and operational records about what happened. To assess the success of post-disaster recovery efforts, it is important to consider the changes that have occurred. The examples identified through this review provide some useful tools to achieve this. However, there were no examples identified that made the critical link between program intervention, impact, and outcomes. The next section of this paper discusses some useful methods for outcomes focused evaluations.

Evaluation frameworks

This review demonstrates a clear need for more consistent and comprehensive evaluations of post-disaster recovery that link interventions to impact and outcomes. Currently, post-disaster recovery in Australia and New Zealand takes a significant amount of public expenditure. As discussed above, recovery from the Christchurch earthquake is predicted to take the equivalent of about 20 percent annual GDP for New Zealand, while the Australian Government continues to expend considerable resources on recovery from previous disaster events (Productivity Commission 2014; The Treasury 2013). Yet it is unclear whether this spend is justified and whether the resources are being efficiently allocated to deliver effective outcomes that support community recovery.

There are a number of reasons why evaluations are not conducted, including unclear definition of post-disaster recovery, the types of activities and actions that characterise this phase of disaster management, and what ‘success’ looks like. The New Zealand and Australian government adopt a holistic framework for post-disaster recovery that places community at the core of the recovery process. However, what this means in terms of outcomes for the community is challenging to define.

These challenges to evaluating post-disaster recovery, and lack of outcomes-focused evaluations identified from the review, provide strong justification for the development of a national monitoring and evaluation framework. The framework should provide practitioners with a clear understanding of what post-disaster recovery is and what success may look like. It should also provide guidance on timing and methodology. Critically, the framework needs to adopt a methodology to ensure practitioners can connect interventions with impacts and outcomes, and determine whether resources are being allocated efficiently to support community recovery.

There are a number of methodologies and approaches for program evaluation that may be useful. Theory of Change (TOC) is a method used for evaluations that maps out the short and mid-term outcomes that are expected to result in the achievement of the long-term goals of the interventions (Weiss 1995). The approach is most useful for complex, long term social issues and may be linked to logic models that explicitly connect the programs inputs and actions with the outputs and desired outcomes. TOC may be appropriate for the complex policy environment of post-disaster recovery. Connell and Kubisch (1998: 1) promote TOC as an appropriate methodology to evaluate comprehensive community initiatives with

  • multiple strands (economic, political, and social), which operate at many levels (community, institutional, personal network, family, and individual), are co-constructed in a collaborative process by diverse stakeholders, and evolve over the course of the initiative.

For post-disaster recovery practitioners, developing a TOC and associated logic models could be a part of the planning process for the intervention and may be a useful framework for evaluations.

Importantly, the results of an evaluation should be useful and able to drive changes that will improve the policy intervention. Goni (2012) highlight this issue in an analysis of evaluations for public expenditure management in OECD countries. Of the seven countries studied, Goni (2012) found that although most implemented public expenditure evaluations, there was little evidence of these being used to inform spending management behaviour. The author suggests that evaluations need to be tailored to the complexity of public management context to increase their usefulness for decision making.

For post-disaster recovery, it is imperative that government spending is efficient, with resources allocated to interventions that are effective for supporting community recovery. Evaluating spending efficiency is one approach that could be used to determine whether resources are being efficiently converted to outcomes, and to identify how resources could be better allocated. This may be integrated into a TOC. Brecher et al. (2005) demonstrate how an evaluation can be used to test the TOC and help improve the delivery of a program or intervention. The authors conducted a fiscal analysis evaluation during the interim period of a health initiative and use the findings to redefine the goals of resource allocation and modify the TOC to better achieve the desired outcomes. Seifert and Nieswand (2014) demonstrate how spending efficiency evaluation techniques can identify inefficiencies and areas of improvement in local government spending. Efficiency analysis is used as a benchmarking approach to compare the transformation of one unit of resource input into output. Using this approach, unit-level inefficiencies in government departments in metropolitan France, and the factors that led to these inefficiencies, were identified. A similar approach was used by Afonso and Fernandez (2007) to measure local government spending in Lisbon. The results of their spending efficiency evaluation indicate that the same level of output could be achieved using one third less resources.

In a post-disaster recovery evaluation, incorporating spend efficiency using a spend efficiency tool may allow practitioners to identify which interventions are delivering most benefit for community recovery relative to input. The benefit to community may be determined as meeting their needs (commonly identified in a needs assessment immediately following a disaster) and progress towards recovery. In addition, adopting spend efficiency analysis in a national monitoring and evaluation framework may allow for comparison between post-disaster recovery interventions from comparable disaster events. This could be highly beneficial for identifying ‘best practice’ interventions that meet community needs and support community recovery. However, given the complexity of post-disaster recovery, measuring spend efficiency may present a significant challenge. Further research on the merits of spend analysis tools is required.

An important element of an evaluation is understanding how decisions are made about the allocation of resources. Ideally, resources are allocated efficiently to address the actual needs that drove the intervention, and deliver the desired outcomes. In reality, however, there are numerous factors that influence resource allocation that may or may not reflect the actual needs. Hajnal and Trounstine (2010) examine what influences local decision making by analysing government spending patterns from a range of nationally representative studies. They find that economic constraints are critical influences in decision making on resource allocation. The authors further note that, in addition to actual needs, ‘redistributional, allocational, and developmental spending is also strongly influenced by political imperatives [and] institutional constraints’ (Hajnal and Trounstine 2010, 1130). For post-disaster recovery, there are likely to be similar influences on how and where resources are allocated that are independent of the actual needs of the community. Without evaluations being consistently conducted, however, it is unclear what these influences are and whether they are impeding effective post-disaster recovery.

Post-disaster recovery: A complex policy environment

Recovery from a disaster event is a complex process. The disasters reviewed in this paper included multiple levels of government, non-government organisations, and volunteers, and covered social, economic, built, and environment domains. Community involvement in the recovery process was identified in the literature as a key to success. However, this too adds to the complexity of the recovery process and attempts at evaluating it. There is a need to conduct these evaluations in a consistent manner to learn from experience and be able to target resources in disaster recovery.

A national framework for disaster recovery is an important step in achieving this. This will provide a useful starting place in knowing where to begin in this complex policy and data collection environment. It will also help achieve consistency in the evaluations undertaken and help lift post-disaster recovery evaluations from being a small addition to response and relief reviews to being a meaningful evaluation of outcomes. It may also help ensure that evaluations are conducted across the whole recovery effort, rather than focusing on specific sub elements or the disaster management process. The theory of change method is a useful approach for developing the overarching framework, as it is appropriate for the complex, long-term, multi-stakeholder nature of post-disaster recovery.

To develop a theory of change, measures of the activities and outcomes of the intervention are required. This typology and review may be used to identify disaster events that provided useful methods to measure activities and outcomes, and thus assist with building a theory of change and the development of a national framework. Useful examples include disaster events where a post-disaster recovery evaluation has been undertaken; the evaluation is outcomes focused (with sufficient attention to process); and it looks at the social, economic, built, and environmental dimensions of recovery. The holistic, outcomes focused evaluations by the Canterbury Earthquake Recovery Authority (2014) and Plyer et al. (2013) are some examples that may be useful for further investigation.

In addition, there is merit in further investigation of how spending analysis evaluation tools could be applied to the post-disaster recovery context. Given the significant amount of public expenditure on post-disaster recovery, efficiency or fiscal analysis would help identify areas of improvement that could increase the efficiency of the public expenditure. Government intervention for post-disaster recovery is critical for helping communities following a disaster event, and knowledge built through evaluation practice ensures resources are used in a manner that best achieves this aim.

The following points summarise the implications of this review for policy and practice:

  • Post-disaster recovery evaluation, conducted within an agreed monitoring and reporting framework, should be conducted for disaster events. Currently, there is a greater emphasis on reviewing the immediate response and relief efforts.

  • Although process evaluations are useful and important, they do not illuminate the full impact or outcomes of the post-disaster recovery effort. Outcomes focused evaluations can assess the effectiveness, efficiency and appropriateness of post-disaster interventions.

  • Theory of change methodology can be used to develop evaluation approaches suited to the complex policy and intervention environment of post-disaster recovery.

  • Outcomes based evaluations identified such as those deployed by the Canterbury Earthquake Recovery Authority (2014) and Plyer et al. (2013) may prove useful examples to identify measures of outcomes to inform a theory of change for post-disaster recovery evaluation.

Appendix 1

Table 4

The disaster characteristics and evaluations found in the grey literature by type of disaster

Location

Date of event

Immediate Social Impact (# casualties

Estimated economic cost (insured costs normalised to current)

Domains of post-disaster recovery activities

Community involvement in recovery planning

Agencies involved in recovery

Evals identified

Process, outcomes focused or mixed

Bushfire

North East and East Gippsland, Victoria

2003

71

$12 million

Built, economic and environmental

Community consultation on significance of assets to be restored, informed of the recovery effort, education and communication

State and local gov, and NGOs

1

Mixed

Victoria

2009

173

$1.07 billion

Social, built, economic and environmental

Community reference groups established to identify recovery priorities

Federal, state, local gov and NGOs

6

Mixed (x3)

Process (x3)

ACT

2003

4

$660 million

Social, built, economic and environmental

Community Expert Reference Group

Federal, territory gov and NGOs

2

Process (x2)

Coonabaraban, NSW

2013

0

$35 million

Social, built, economic and environmental

Community newsletter

Federal, state, local gov, NGOs/volunteers

1

Process

Dunalley, TAS

2013

1

$89 million

Social, built and economic

Workshops and informal community engagement

Federal, state local gov, NGOs/volunteers

1

Process

Eyre Peninsula, South Australia

2005

9

$41 million

Social, built and economic

Community consultation and community representatives on the workforce recovery committee

Federal, state, local gov, NGOs/volunteers

1

Process

Margaret River, WA

2011

0

$53.5 million.

Social, built and environmental

Community meetings

Federal, state, local gov, NGOs/volunteers

2

Process (x1)

Mixed (x1)

Blue Mountains, NSW

2013

2

$183 million

Social, built, economic and environmental

Online survey and forums to identify community priorities

Federal, state, local gov, NGOs/volunteers

1

Mixed

Perth Hills, WA

2014

1

$15 million

Social and built

Local Recovery Coordination Committee meetings including community

Federal, state, local gov, NGOs/volunteers

1

Process

Victoria

2014

0

NA

Social, built and economic

Engagement of community in recovery planning

State and local gov, volunteers

1

Process

Widespread Victoria

2006

4

$28 million

Social, built, economic and environmental

Community meetings

Federal, state, local gov, NGOs/volunteers

1

Process

California, USA

2003

15

$27 million (US)

Built and environmental

NA

All tiers of gov and NGOs/Volunteers

2

Process (x1)

Outcomes (x1)

Criminal Act

NY, USA

2001

168

NA

Social, built and economic

Community consultation and engagement

All tiers of gov and NGO’s/Volunteers

2

Process (x1)

Outcomes (x1)

Bali

2002

26

NA

Social, built and economic

NA

Federal gov (Australia and Indonesia), NGOs

1

Mixed

London

2005

52

NA

Social

NA

Federal and local gov, NGOs

2

Process (x1)

Outcomes (x1)

Oklahoma,

1995

168

$652 million (US)

Social and built

Community engagement

All tiers of gov and NGOs/Volunteers

1

Process

Northern Ireland

1998

29

NA

Social

Community-led support group

Federal and local gov, NGOs

1

Process

Cyclone

North QLD

2006

1

$609 million

Social, built, economic and environmental

NA

Federal, state, local gov, NGO’s/volunteers

1

Process

QLD

2011

1

$800 million

Social, built, economic and environmental

Community forums

Federal, state, local gov, NGOs/volunteers

1

Mixed

QLD and NSW

2013

6

$1.10 billion

Social, built, economic and environmental

NA

Federal, state, local gov, NGOs/volunteers

1

Process

Hurricane

New Orleans, USA

2005

1,833

$135 billion (US)

Social, built, economic and environmental.

NA

All tiers of gov and NGOs/Volunteers

2

Outcomes (x2)

New York, USA

2012

117

$700 million (US)

Social, built, economic and environmental

Community engagement

All tiers of gov and NGOs/Volunteers

1

Process

Earthquake

Christchurch

2011

185

$370 million

Social, built, economic and environmental

Community workshops Online forum

Federal, local gov, NGOs/volunteers

1

Outcomes

Japan

2011

15,889

$235 billion (US)

Social, built, economic and environmental

NA

National and local gov, NGOs

1

Outcomes

Sumatra

2009

1117

NA

Built and economic

Community survey

National gov and NGOs

1

Outcomes

Flood

Gawler River, South Australia

2005

0

$40 million

Social, economic and environmental

Community engagement, participation and empowerment

State, local gov and NGOs

2

Mixed (x2)

Gippsland, Victoria

2007

1

$18 million

Social, built, economic and environmental

Raising community awareness of recovery program

State, local gov and NGOs

1

Mixed

Mid and North Coast, New South Wales

2009

0

$40 million

Social, built, economic and environmental

Elected representatives of community on recovery committee

State, local gov and NGOs

1

Process

QLD

2010-11

33

$2.38 billion

Social, built, economic and environmental

Community ballot and consultation

Federal, state, local gov, NGOs/vol

1

Process

Victoria

2011

1

$126 million

Social, built and economic

Community consultation sessions

Federal, state, local gov, NGOs/vol

4

Process (x4)

Alberta, Canada

2013

4

$6 billion (CA)

Social, built, economic and environmental

Community engagement

Provincial gov and NGOs

2

Process (x2)

Tsunami

Indian Ocean

2004

165,945

$10 billion (US)

Social, built, economic and environmental

NA

Federal gov (Australia and Indonesia) and NGOs

3

Process (x1)

Outcomes (x2)

Samoa, American Samoa, Tonga

2009

144

$150 million

Social, built, economic and environmental.

Consultation with communities over re-settlement

International gov, federal gov (Samoa), NGOs

2

Process

Mixed

Storm surge

UK

2013-14

0

NA

Built Economic

NA

Federal and local gov

1

Process

Table 5

Details of the evaluations undertaken for each disaster event by evaluation type

Who conducted the evaluation

Data collection method

Evaluation focus

Reference

Process

 

Outcomes

 

After action review/debrief

Government department

Experience of operational staff

Community debrief

Process of implementation of recovery actions

Social and built

 

 

Warrumbungle Shire Council (2013)

Community representatives

Personal experiences

Communication of processes and implementation of programs

Social

 

 

Community Recovery Committee (2011)

Community recovery report

Recovery team

Survey of community

Community perceptions of process of recovery

Social and environmental

Preparedness for next event

Social

Bushfire Recovery Team (2012)

Government review

Government department

Personnel experience and review of operational records and reports

Replacement and repair of assets; implementation of restoration/environmental management programs; $ support for farmers, set up and use of support programs; implementation of education programs; use of communication tools

Social, built, economic and environmental

Water quality monitoring, in-stream fauna, relationships between government agencies and special interest groups

Environmental

Victorian Government Department of Sustainability and Environment (2005)

Recovery authority

Personal experiences

The programs delivered and ongoing activities

Social, built, economic and environmental

 

 

VBBRA (2011)

Commission of inquiry

Review of operational documents, witness accounts

Review of the policy/planning framework and governments management of recovery

Social and built

 

 

VBRC (2010)

Senior recovery official

Community consultation and secondary data

Access to health services, support for business owners, distribution of support funds, implementation of environmental recovery programs

Social, economic, and environmental

Rebuilding, regaining previous state of mental wellbeing, performance at school, rate of anti-social behaviour, return of tourism

Built, social and economic

Hubbard (2014)

Recovery authority

Review of operational documents

Delivery of programs

Social, built, economic and environmental

 

 

ACT Bushfire Taskforce Secretariat (2003)

Government department

Review of operational documents, witness accounts

Review of transition from response to recovery, leadership and establishment of programs, committees and funding for recovery.

Social and built

 

 

Tasmanian Bushfire Inquiry (2013)

Government department

Interviews with stakeholders and document review

Assessment of whether the recovery process aligned with disaster recovery principles

Social, built, economic and environmental

 

 

State Recovery Committee (2005)

Government department

Agency experiences

Progress towards implementing programs and lessons learned from experience

Social and environmental

 

 

Noetic Solutions Pty Limited (2012)

Government department

Review of reports and records, and interviews with key personnel and volunteers

Transition from response to recovery, implementation of recovery programs

Social and built

 

 

State Emergency Management Committee (2014)

Government department

Data and document review, consultation with key stakeholders, survey of community

Management of recovery and program delivery by local governments

Social, built, economic and environmental

 

 

Inspector-General for Emergency Management (2016)

Government department

Experience of taskforce

Funding allocation and establishment of recovery programs/initiatives

Social, built and economic

 

 

Ministerial Taskforce on Bushfire Recovery (2006)

Government department

Review of funding documentation

Review of legitimacy of government spending

Economic

 

 

U.S. Government (2006)

Australian Treasury

Review of survey data (secondary)

Review of funding processes

Economic

Economic recovery as GDP growth

Economic

Lucich et al. (2006)

Government department

Review of operational documents, testimony from survivors

Review of processes in place to assist social recovery

Social

 

 

London Assembly (2006)

Government department

Review of operational documents

Review of government planning and resource support

Social

 

 

Oklahoma Department of Civil Emergency Management (n.d.)

Commission of inquiry

Formal minutes and witness accounts

Treatment of victims

Social

 

 

Capita Health and Wellbeing (2003)

Government department

Experience of taskforce/agencies

Review of recovery organisation and program implementation

Social, built, economic and environmental

 

 

Queensland Government (2007)

Government department

Review of operational documents and organisations experience

Review of processes to secure the port in New York, establishment of relationships and trust between agencies.

Social

 

 

Sturgis et al. (2014)

Recovery authority

Survey and review of secondary survey data

 

 

Maintenance of education sector, housing, economic wellbeing, health, personal safety, social connectedness, mental wellbeing

Social, built, economic and environmental

Canterbury Earthquake Recovery Authority (2014)

Government department with assistance from independent researchers

Interviews with community and those involved in the recovery effort

Recovery processes including stakeholder commitment, engagement, participation, communication and administration processes

Social and economic

Indicators of community development including informed community, healthy community, community cohesion and economic recovery

Social and economic

Department for Families and Communities (n.d.)

Government department

Review of records and operational data, fauna and vegetation surveys, archaeological surveys

Programs implemented to address recovery needs

Social, economic and environmental

Platypus return to creeks and vegetation growth

Environmental

Victorian Government of Sustainability and Environment & Parks (2010)

Government department

Experiences of the recovery committee and those involved in recovery

Implementation of programs

Social, built and economic

 

 

Recovery Coordinator (2009)

Commission of inquiry

Review of operational documents, witness accounts

Review of planning instruments and government response

Social and built

 

 

Queensland Flood Commission of Inquiry (2012)

Commission of inquiry

Review of legislation and policies

Assessment of planning and mitigation strategies, allocation of grants and community engagement

Social and built

 

 

Comrie (2011)

Local council

Independently facilitated workshops

Recovery processes that worked well or not

Social, built and economic

 

 

Buloke Shire Council (2011)

Government department

Audit of government programs

Functioning of committees and adequacy of programs for covering social, economic and environmental needs

Social, built, economic and environmental

 

 

Victorian Auditor General’s Office (2013)

Government department

Engagement with public and water experts

Progress towards fulfilling recommended steps for recovery

Social and environmental

 

 

WaterSMART (2014)

Joint Standing Committee on Foreign Affairs, Defence and Trade

Witness testimonies

Review of funding allocation

Social and built

 

 

Commonwealth of Australia (2006)

Government department

Review of government programs

Provision of financial relief to communities, businesses and rebuilding efforts

Social, built and economic

 

 

Department for Communities and Local Government. (2014)

Independent evaluation

External consultancy

Documentation of the experiences of those involved

Activation of government agencies and implementation of recovery plans/programs

Social, built, economic and environmental

 

 

PricewaterhouseCoopers (2010)

External consultancy

information review, consultation with staff and internal stakeholders, external stakeholders and grant recipients

Delivery of the arts programs, number of grants recipients

Social

Increased sense of community, improved confidence

Social

Fisher and Talve (2011)

Independent research

Survey and interviews with community

Implementation of programs, communication strategies and involvement of the community

Social

Mental health outcomes

Social

Camilleri et al. (2007)

Independent research

Interviews with key stakeholders and analysis of secondary data

Services used by respondents

Social

Perceived effectiveness for improving psychosocial recovery

Social

Rich et al (2014)

Independent research

Review of procedures

Forest restoration practices

Social and environmental

 

 

Farm and Home Advisor’s Office (2007)

Independent research

Field surveys

 

 

Environmental indicators

Environmental

Robichaud et al. (2010)

Independent research

Long-term clinical assessments

 

 

Mental and physical health

Social

Mount Sinai Medical Center (2011)

Independent research

Patient throughput and screening process and survey

 

 

Indicators for mental health

Social

Brewin et al (2009)

Independent research

Interviews, focus groups, survey of council personnel

Response of authorities for restoring essential services, involvement of community in recovery and grant distribution.

Built

Recovery of population, tourism trade, agricultural sector and employment

Social and economic

Regional Australia Institute (2013)

External consultancy

Review secondary survey data

 

 

Economic recovery, inclusion (in the labour market and housing), quality of life and sustainability

Social, built, economic and environmental

Plyer et al. (2013)

External consultancy

Review secondary survey data

 

 

Labour market outcomes

Economic

RAND Labor and Population (2010)

Independent research

Secondary data review, key informant interviews, focus groups, an on-line survey with National Societies, field visits to the prefectures and municipalities affected by the disaster, and observation

 

 

Efficiency, effectiveness, relevance and appropriateness of the Red cross recovery interventions

Social, built, economic and environmental

Babe et al. (2013)

External consultancy

Stakeholder survey

 

 

Health impacts, effectiveness of re-housing

Social and built

Frankenberg et al. (2014)

Non-government review

Queensland Farmers Federation

Review of reports and records

Funding allocation and establishment of recovery programs/initiatives

Social, built, economic and environmental

 

 

Queensland Farmers Federation (2014)

NGO

Survey, observations and secondary data analysis

 

 

Provision of appropriate shelter, economic stimulus through job creation

Built and economic

International Federation of Red Cross and Red Crescent Society, United Nations High Commissioner for Refugees and UN-Habitat (2010)

NGO

Not defined

Programs implemented and funding allocated

Economic

Economic recovery assessed through horticultural revenue

Economic

Virginia Horticultural Centre South Australia (2008)

NGO

Roundtables with community sector organisations

Establishment of processes to aid psychosocial recovery, provision of housing support and appropriate funding allocation.

Social

 

 

Victorian Council of Social Service (2011)

NGO

Personal communications

Steps taken for recovery and allocation of funding

Social and built

 

 

Red Cross (n.d.)

NGO

Literature review, surveys and field survey

 

 

Health and child protection

Social

UNICEF (2009)

NGO

experience of those involved in recovery efforts

Activities undertaken, people reached by the program, funding allocation

Social and built

 

 

Oxfam (2010)

NGO

experience of those involved in recovery efforts

Delivery and participation in programs

Social and built

Food production

Social

International Federation of Red Cross and Red Crescent Society (2011)

Notes

[1] The Australian Emergency Management Knowledge Hub is an online resource that provides a catalogue of disasters that have occurred in Australia and nearby regions (Australian Emergency Management Institute, n.d.). The resource provides key facts and links to sources of additional information.

[2] The complete typology is available in a Microsoft Excel spreadsheet. Please contact the authors to request a copy.

References


  1. 1
    ACT Bushfire Taskforce Secretariat 2003. Submission to the inquiry into the operational response to the January bushfires. Viewed 17 November 2015
    [URL]
  2. 2
    Afonso, A and Fernandez, S 2007. Measuring Local Government Spending Efficiency: Evidence for the Lisbon Region. Regional Studies, 40(1), 39-53.
    [CROSSREF]
  3. 3
    Althaus, C, Bridgman, P, and Davis, G 2007. The Australian Policy Handbook (4 Edition). Allen & Unwin, Sydney.
  4. 4
    Archer, F, Roberts, F, McArdle, D, and Spencer, C 2015. Literature review: What does good or successful recovery look like? Monash University, Injury Research Institute. Australian and New Zealand Emergency Management Committee, Melbourne.
  5. 5
    Australasian Evaluation Society. (2010) AES Guidelines for the Ethical Conduct of Evaluations. Viewed 17 November 2015,
    [URL]
  6. 6
    Australian Audit Office 2015. Administration of the Natural Disaster Relief and Recovery Arrangements by Emergency Management Australia. Australian National Audit Office, Canberra.
  7. 7
    Australian Emergency Management Institute. (n.d.) Disaster Information. Viewed 20 October 2015,
    [URL]
  8. 8
    Babe, M, Horekens, J, Kagmi, S, Stansberry, M and Tochibayashi, N 2013. Evaluation of the JRCS and IFRC recovery and rehabilitation interventions after the great east japan earthquake and tsunami of 11 March 2011. Japanese Red Cross Society, viewed 17 November 2015,
    [URL]
  9. 9
    Bovens, M 2005. Public Accountability. In E Ferlie, L E Lynne Jr and C Pollitt (eds) The Oxford Handbook of Public Management, Oxford University Press, Oxford
  10. 10
    Brewin, C R Fuchkan, N, and Huntley, Z 2009. Evaluation of the NHS Trauma Response to the London Bombings (MRD 12/98). Final Report to the Department of Health Forensic Mental Health R & D Programme. Department of Health, London.
  11. 11
    Brecher, C, Silver, D, Searcy, C, and Weitzman, BC 2005. Following the money: Using expenditure analysis as an evaluation tool. American Journal of Evaluation, 26, 166–188.
    [CROSSREF]
  12. 12
    Buloke Shire Council 2011. 2011 Flood Recovery Evaluation. Viewed 20 November 2015,
    [URL]
  13. 13
    Bushfire Recovery Team 2012. Community Recovery Report: Eight Months on from the November 2011 Margaret River Bushfires. Viewed 17 November 2015,
    [URL]
  14. 14
    Camilleri, P, Healy, C, Macdonald, E, Nicholls, S, Sykes, J, Winkworth, G and Woodward, M 2007. Recovering from the 2003 Canberra bushfire: A work in progress. Canberra. Australian Catholic University, Canberra
  15. 15
    Canterbury Earthquake Recovery Authority 2014. Canterbury Wellbeing Index June 2014. Canterbury Earthquake Recovery Authority, Christchurch.
  16. 16
    Capita Health and Wellbeing 2003. Evaluation of Health and Social Services for Victims of the Conflict. Viewed 22 November 2015,
    [URL]
  17. 17
    Chouinard, JA 2013. The Case for Participatory Evaluation in an Era of Accountability, American Journal of Evaluation, 34(2), 237-253.
    [CROSSREF]
  18. 18
    Commonwealth of Australia 1998. Australian Emergency Management Glossary. Canberra: Emergency Management Australia.
  19. 19
    Commonwealth of Australia 2006. Australia’s Response to the Indian Ocean Tsunami. Viewed 17 November 2015,
    [URL]
  20. 20
    Commonwealth of Australia 2011. Community Recovery Handbook 2. Australian Emergency Management Institute, Canberra.
  21. 21
    Community Recovery Committee 2011. Lessons Learned by Community Recovery Committees of the 2009 Victorian Bushfires. Viewed 17 November 2015
    [URL]
  22. 22
    Comrie, N 2011. Review of the 2010–11 Flood Warnings & Response. State of Victoria, viewed 20 April 2015,
    [URL]
  23. 23
    Connell, J and Kubisch, A 1998. Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects and Problems. In K Fulbright-Anderson, A Kubisch and J Connell (eds) New Approaches to Evaluating Community Initiatives, vol. 2, Theory, Measurement, and Analysis. Aspen Institute, Washington, DC.
  24. 24
    DeLuca, TH, Aplet, GH, Wilmer, B and Burchfield, J 2010. The Unknown Trajectory of Forest Restoration: A Call for Ecosystem Monitoring. Journal of Forestry, 108(6), 288-295.
  25. 25
    Department for Communities and Local Government 2014. Winter 2013/14 severe weather recovery progress report. Viewed 17 November 2015,
    [URL]
  26. 26
    Department for Families and Communities n.d. Gawler River Flood Recovery: Virginia Community Development Project Report. Government of South Australia, Adelaide
  27. 27
    Dooley, LN 2002. Case Study Research and Theory Building. Advances in Developing Human Resources, 4(3), 335.
    [CROSSREF]
  28. 28
    Dufty, D 2013. Evaluating Emergency Management after an Event: Gaps and Suggestions. Australian Journal of Emergency Management 28, 4.
  29. 29
    Farm and Home Advisor’s Office 2007. Wildfire Preparedness and Recovery in San Diego County: A Review and Analysis White Paper of Data and Research Studies Relevant to Wildfire. Viewed 22 November 2015,
    [URL]
  30. 30
    FEMA [Federal Emergency Management Agency] 2011. National Disaster Recovery Framework. Viewed 21 May 2015, [URL]
  31. 31
    Fisher, N and Talve, A 2011. The role of arts in rebuilding community. NSF Consulting, viewed 17 November 2015,
    [URL]
  32. 32
    Frankenberg, E, Thomas, D and Friedman, J 2014. The Study of the Tsunami Aftermath and Recovery (STAR): Resilience and Recovery in Sumatra after the 2004 Indian Ocean Tsunami. Viewed 17 November 2015,
    [URL]
  33. 33
    Goni, EZ 2012. Evaluation for public expenditure management: international experiences in some OECD countries. Presupuesto y Gasto Público, 68, 53-80.
  34. 34
    Hajnal, ZL, and Trounstine, J 2010. Who or what governs? The effects of economics, politics, institutions, and needs on local spending. American Politics Research, 38(6), 1130-1163.
    [CROSSREF]
  35. 35
    Hawkins, R, and Maurer, K 2010. Bonding, Bridging and Linking: How social capital operated in New Orleans following Hurricane Katrina. British Journal of Social Work, 40, 1777-1793.
    [CROSSREF]
  36. 36
    Hubbard, B 2014. Report to the Victorian Government of the 2009 Bushfire-Affected Communities-Five Year Consultation. Viewed 1 December 2015,
    [URL]
  37. 37
    Inspector-General for Emergency Management 2016. Review of community recovery following the 2013–14 Victorian bushfires. State of Victoria, viewed 17 November 2015
    [URL]
  38. 38
    International Federation of Red Cross and Red Crescent Society, United Nations High Commissioner for Refugees and UN-Habitat 2010. Shelter Projects 2010. IFRC, UNHCR and UN-Habitat, pp. 38-50
  39. 39
    Leadbeater, A 2013. Community Leadership in Disaster Recovery: A Case Study. Australian Journal of Emergency Management, 28(3), 41-47.
  40. 40
    London Assembly 2006. Report of the 7 July Review Committee. Viewed 22 December 2015,
    [URL]
  41. 41
    Lucich, M, Dal Bon, N and Houston, E 2006. The Indonesian Economy after the Boxing Day Tsunami and Treasury’s Role in the Government Partnership Fund. Viewed 17 November 2015
    [URL]
  42. 42
    Ministerial Taskforce on Bushfire Recovery 2006. 2006 Report from the ministerial taskforce on bushfire recovery. State of Victoria, viewed 17 November 2015,
    [URL]
  43. 43
    Ministry for Civil Defence & Emergency Management 2005. Focus on Recovery: A Holistic Framework for Recovery in New Zealand. Ministry for Civil Defence & Emergency Management, Wellington.
  44. 44
    Ministry for Civil Defence & Emergency Management n.d. The 4Rs: Reduction, readiness, response and recovery. Viewed 10 April 2016, [URL]
  45. 45
    Mount Sinai Medical Center 2011. First Long-Term Study of World Trade Center Rescue and Recovery Workers Shows Widespread Health Problems Ten Years after 9-11. Viewed 22 November 2015,
    [URL]
  46. 46
    National Audit Office 2013. Evaluation in Government. National Audit Office, London.
  47. 47
    Noetic Solutions Pty Limited 2012. Report on the Post Incident Analysis of the 2011 Margaret River and Nannup bushfires. State Emergency Management Committee, viewed 17 November 2015,
    [URL]
  48. 48
    NZ Treasury 2011. Economic and Fiscal Impacts of the Canterbury Earthquakes. Viewed 31 March 2016,
    [URL]
  49. 49
    Oklahoma Department of Civil Emergency Management. (n.d.). After Action Report. Oklahoma Department of Civil Emergency Management, Oklahoma.
  50. 50
    Owen, J 2006. Program Evaluation: Forms and Approaches, Allen & Unwin 3rd edition. Crows News.
  51. 51
    Oxfam 2010. From tragedy to recovery: Samoa tsunami response 2009-2010. Oxfam New Zealand, Auckland.
  52. 52
    Plyer, A, Ortiz, E, Horwitz, B and Hobor, G 2013. The New Orleans Index at Eight. The Data Center, New Orleans, viewed 22 November 2015,
    [URL]
  53. 53
    PricewaterhouseCoopers 2010. Effective Disaster Recovery: What Lessons can we learn from Australia’s Black Saturday Bushfires? Viewed 20 November 2015,
    [URL]
  54. 54
    Productivity Commission 2014. Natural Disaster Funding Arrangements-Volume 1. Inquiry Report no.74, Canberra.
  55. 55
    Queensland Farmers Federation 2014. Cyclone Oswald Industry Recovery Initiative viewed 22 November 2015,
    [URL]
  56. 56
    Queensland Flood Commission of Inquiry 2012. Queensland Flood Commission of Inquiry Final Report. Viewed 20 November 2015,
    [URL]
  57. 57
    Queensland Government 2007. The Final Report of the Operation Recovery Task Force. State of Queensland, viewed20 November 2015,
    [URL]
  58. 58
    RAND Labor and Population 2010. The Workforce and Economic Recovery: Effects of Hurricane Katrina. Viewed 20 November 2015,
    [URL]
  59. 59
    Recovery Coordinator 2009. Recovery Coordinator’s Report on the Mid-North and Far-North Coast Flood Recovery May 2009. Viewed 17 November 2015,
    [URL]
  60. 60
    Red Cross n.d. Alberta Floods: One Year Progress Report. Red Cross, viewed 22 April 2015, [URL]
  61. 61
    Regional Australia Institute 2013. From Recovery to Renewal. Viewed 19 November 2015,
    [URL]
  62. 62
    Rich JL, Booth A, Reddy P and Rowlands A 2014. The Step by Step Bushfire Support Service Qualitative Evaluation Report. Centre for Rural and Remote Mental Health, University of Newcastle.
  63. 63
    Robichaud, PR, Elliot, WJ, Wagenbrenner, JW, Lewis, SA, Ashmun, LE, Wohlgemuth, PM and Brown, RE 2010. Continued Evaluation of Post-Fire Recovery and Treatment Effectiveness for Validation of the ERMiT Erosion Model (Paper 41). Viewed 22 November 2015,
    [URL]
  64. 64
    Ryan, R 2014. Presentation of the logic of programs, UTS, unpublished
  65. 65
    Seifert, S and Nieswand, M 2014. What drives intermediate local governments’ spending efficiency: the case of French Départments. Local Government Studies, 40 (5), 766-790.
    [CROSSREF]
  66. 66
    Shkabatur, J 2012. Transparency With(out) Accountability: Open Government in the United States. Yale Law & Policy Review, 31(1), 80-120.
  67. 67
    State Emergency Management Committee 2014. Parkerville Stoneville Mt Helena Bushfire Review. Government of South Australia, viewed 8 December 2015
    [URL]
  68. 68
    State Recovery Committee 2005. Collaboration is Key. Government of South Australia, viewed 24 November 2015,
    [URL]
  69. 69
    Sturgis, LA, Smythe, TC and Tucci, AE 2014. Port Recovery in the Aftermath of Hurricane Sandy: Improving Port Resiliency in the Era of Climate Change. Center for a New American Security, viewed 22 November 2015,
    [URL]
  70. 70
    Tasmanian Bushfire Inquiry 2013. 2013 Tasmanian Bushfires Inquiry: Volume One. Viewed 22 November 2015,
    [URL]
  71. 71
    The Sphere Project 2015. Sphere for Monitoring and Evaluation. The Sphere Project. Geneva
  72. 72
    The Treasury 2013. 2013 Budget: building momentum. Viewed 15 April 2016,
    [URL]
  73. 73
    U.S. Government 2006. An Examination of Federal 9/11 Assistance to New York: Lessons Learned in Preventing Waste, Fraud, Abuse, and Lax Management (20-452). Viewed 22 November 2015,
    [URL]
  74. 74
    UNICEF 2009. Children and the 2004 Indian Ocean Tsunami: Evaluation of UNICEF’s Response in Indonesia (2005-2008). Viewed 22 November 2015,
    [URL]
  75. 75
    VBBRA [Victorian Bushfire Reconstruction and Recovery Authority] 2011. Expert Reference Group Legacy Report. VBRRA, viewed 22 November 2015,
    [URL]
  76. 76
    Victorian Auditor General’s Office 2013. Flood Relief and Recovery. Viewed 22 November 2015,
    [URL]
  77. 77
    Victorian Bushfire Royal Commission 2010. Final Report: Summary. Viewed 20 November 2015,
    [URL]
  78. 78
    Victorian Council of Social Service 2011. Victorian Floods Review. Viewed 20 November 2015,
    [URL]
  79. 79
    Victorian Government Department of Sustainability and Environment 2005. The Recovery Story: The 2003 Alpine Fires. Victorian Government Department of Sustainability and Environment, Melbourne
  80. 80
    Victorian Government of Sustainability and Environment & Parks 2010. 2007 Gippsland Flood/Storm Recovery Program Final Report. State of Victoria Department of Sustainability and Environment & Parks Victoria, Melbourne
  81. 81
    Virginia Horticultural Centre South Australia 2008. Gawler River Flood Recovery Program. Virginia Horticultural Centre South Australia, Virginia
  82. 82
    Warrumbungle Shire Council 2013. Wambelong Redbank Section 44 Bushfire Compilation of Reports. Viewed 20 November 2015,
    [URL]
  83. 83
    WaterSMART 2014. The 2013 Great Alberta Flood: Progress Report on Actions to Mitigate, Manage and Control Flooding. Viewed 20 November 2015,
    [URL]
  84. 84
    Weiss, CH 1995. Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J P Connell, A C Kubisch, L B Schorr, and CH Weiss (ed.s), New approaches to evaluating community initiatives: Concepts, methods and contexts, Vol. 1, Aspen Institute, Washington, DC.
  85. 85
    Yin, RK 1981. The case study crisis: Some answers. Administrative Science Quarterly, 26(1), 58-65.
    [CROSSREF]
  86. 86
    Yin, RK 2003. Case Study Research: Design and Methods 3rd edition. Sage Publications, Thousand Oaks CA.
XML PDF Share

FIGURES & TABLES

Figure 1

The logic of a program and the relationship with key evaluation themes

Source: Ryan (2014)

Full Size   |   Slide (.pptx)

REFERENCES

  1. 1
    ACT Bushfire Taskforce Secretariat 2003. Submission to the inquiry into the operational response to the January bushfires. Viewed 17 November 2015
    [URL]
  2. 2
    Afonso, A and Fernandez, S 2007. Measuring Local Government Spending Efficiency: Evidence for the Lisbon Region. Regional Studies, 40(1), 39-53.
    [CROSSREF]
  3. 3
    Althaus, C, Bridgman, P, and Davis, G 2007. The Australian Policy Handbook (4 Edition). Allen & Unwin, Sydney.
  4. 4
    Archer, F, Roberts, F, McArdle, D, and Spencer, C 2015. Literature review: What does good or successful recovery look like? Monash University, Injury Research Institute. Australian and New Zealand Emergency Management Committee, Melbourne.
  5. 5
    Australasian Evaluation Society. (2010) AES Guidelines for the Ethical Conduct of Evaluations. Viewed 17 November 2015,
    [URL]
  6. 6
    Australian Audit Office 2015. Administration of the Natural Disaster Relief and Recovery Arrangements by Emergency Management Australia. Australian National Audit Office, Canberra.
  7. 7
    Australian Emergency Management Institute. (n.d.) Disaster Information. Viewed 20 October 2015,
    [URL]
  8. 8
    Babe, M, Horekens, J, Kagmi, S, Stansberry, M and Tochibayashi, N 2013. Evaluation of the JRCS and IFRC recovery and rehabilitation interventions after the great east japan earthquake and tsunami of 11 March 2011. Japanese Red Cross Society, viewed 17 November 2015,
    [URL]
  9. 9
    Bovens, M 2005. Public Accountability. In E Ferlie, L E Lynne Jr and C Pollitt (eds) The Oxford Handbook of Public Management, Oxford University Press, Oxford
  10. 10
    Brewin, C R Fuchkan, N, and Huntley, Z 2009. Evaluation of the NHS Trauma Response to the London Bombings (MRD 12/98). Final Report to the Department of Health Forensic Mental Health R & D Programme. Department of Health, London.
  11. 11
    Brecher, C, Silver, D, Searcy, C, and Weitzman, BC 2005. Following the money: Using expenditure analysis as an evaluation tool. American Journal of Evaluation, 26, 166–188.
    [CROSSREF]
  12. 12
    Buloke Shire Council 2011. 2011 Flood Recovery Evaluation. Viewed 20 November 2015,
    [URL]
  13. 13
    Bushfire Recovery Team 2012. Community Recovery Report: Eight Months on from the November 2011 Margaret River Bushfires. Viewed 17 November 2015,
    [URL]
  14. 14
    Camilleri, P, Healy, C, Macdonald, E, Nicholls, S, Sykes, J, Winkworth, G and Woodward, M 2007. Recovering from the 2003 Canberra bushfire: A work in progress. Canberra. Australian Catholic University, Canberra
  15. 15
    Canterbury Earthquake Recovery Authority 2014. Canterbury Wellbeing Index June 2014. Canterbury Earthquake Recovery Authority, Christchurch.
  16. 16
    Capita Health and Wellbeing 2003. Evaluation of Health and Social Services for Victims of the Conflict. Viewed 22 November 2015,
    [URL]
  17. 17
    Chouinard, JA 2013. The Case for Participatory Evaluation in an Era of Accountability, American Journal of Evaluation, 34(2), 237-253.
    [CROSSREF]
  18. 18
    Commonwealth of Australia 1998. Australian Emergency Management Glossary. Canberra: Emergency Management Australia.
  19. 19
    Commonwealth of Australia 2006. Australia’s Response to the Indian Ocean Tsunami. Viewed 17 November 2015,
    [URL]
  20. 20
    Commonwealth of Australia 2011. Community Recovery Handbook 2. Australian Emergency Management Institute, Canberra.
  21. 21
    Community Recovery Committee 2011. Lessons Learned by Community Recovery Committees of the 2009 Victorian Bushfires. Viewed 17 November 2015
    [URL]
  22. 22
    Comrie, N 2011. Review of the 2010–11 Flood Warnings & Response. State of Victoria, viewed 20 April 2015,
    [URL]
  23. 23
    Connell, J and Kubisch, A 1998. Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects and Problems. In K Fulbright-Anderson, A Kubisch and J Connell (eds) New Approaches to Evaluating Community Initiatives, vol. 2, Theory, Measurement, and Analysis. Aspen Institute, Washington, DC.
  24. 24
    DeLuca, TH, Aplet, GH, Wilmer, B and Burchfield, J 2010. The Unknown Trajectory of Forest Restoration: A Call for Ecosystem Monitoring. Journal of Forestry, 108(6), 288-295.
  25. 25
    Department for Communities and Local Government 2014. Winter 2013/14 severe weather recovery progress report. Viewed 17 November 2015,
    [URL]
  26. 26
    Department for Families and Communities n.d. Gawler River Flood Recovery: Virginia Community Development Project Report. Government of South Australia, Adelaide
  27. 27
    Dooley, LN 2002. Case Study Research and Theory Building. Advances in Developing Human Resources, 4(3), 335.
    [CROSSREF]
  28. 28
    Dufty, D 2013. Evaluating Emergency Management after an Event: Gaps and Suggestions. Australian Journal of Emergency Management 28, 4.
  29. 29
    Farm and Home Advisor’s Office 2007. Wildfire Preparedness and Recovery in San Diego County: A Review and Analysis White Paper of Data and Research Studies Relevant to Wildfire. Viewed 22 November 2015,
    [URL]
  30. 30
    FEMA [Federal Emergency Management Agency] 2011. National Disaster Recovery Framework. Viewed 21 May 2015, [URL]
  31. 31
    Fisher, N and Talve, A 2011. The role of arts in rebuilding community. NSF Consulting, viewed 17 November 2015,
    [URL]
  32. 32
    Frankenberg, E, Thomas, D and Friedman, J 2014. The Study of the Tsunami Aftermath and Recovery (STAR): Resilience and Recovery in Sumatra after the 2004 Indian Ocean Tsunami. Viewed 17 November 2015,
    [URL]
  33. 33
    Goni, EZ 2012. Evaluation for public expenditure management: international experiences in some OECD countries. Presupuesto y Gasto Público, 68, 53-80.
  34. 34
    Hajnal, ZL, and Trounstine, J 2010. Who or what governs? The effects of economics, politics, institutions, and needs on local spending. American Politics Research, 38(6), 1130-1163.
    [CROSSREF]
  35. 35
    Hawkins, R, and Maurer, K 2010. Bonding, Bridging and Linking: How social capital operated in New Orleans following Hurricane Katrina. British Journal of Social Work, 40, 1777-1793.
    [CROSSREF]
  36. 36
    Hubbard, B 2014. Report to the Victorian Government of the 2009 Bushfire-Affected Communities-Five Year Consultation. Viewed 1 December 2015,
    [URL]
  37. 37
    Inspector-General for Emergency Management 2016. Review of community recovery following the 2013–14 Victorian bushfires. State of Victoria, viewed 17 November 2015
    [URL]
  38. 38
    International Federation of Red Cross and Red Crescent Society, United Nations High Commissioner for Refugees and UN-Habitat 2010. Shelter Projects 2010. IFRC, UNHCR and UN-Habitat, pp. 38-50
  39. 39
    Leadbeater, A 2013. Community Leadership in Disaster Recovery: A Case Study. Australian Journal of Emergency Management, 28(3), 41-47.
  40. 40
    London Assembly 2006. Report of the 7 July Review Committee. Viewed 22 December 2015,
    [URL]
  41. 41
    Lucich, M, Dal Bon, N and Houston, E 2006. The Indonesian Economy after the Boxing Day Tsunami and Treasury’s Role in the Government Partnership Fund. Viewed 17 November 2015
    [URL]
  42. 42
    Ministerial Taskforce on Bushfire Recovery 2006. 2006 Report from the ministerial taskforce on bushfire recovery. State of Victoria, viewed 17 November 2015,
    [URL]
  43. 43
    Ministry for Civil Defence & Emergency Management 2005. Focus on Recovery: A Holistic Framework for Recovery in New Zealand. Ministry for Civil Defence & Emergency Management, Wellington.
  44. 44
    Ministry for Civil Defence & Emergency Management n.d. The 4Rs: Reduction, readiness, response and recovery. Viewed 10 April 2016, [URL]
  45. 45
    Mount Sinai Medical Center 2011. First Long-Term Study of World Trade Center Rescue and Recovery Workers Shows Widespread Health Problems Ten Years after 9-11. Viewed 22 November 2015,
    [URL]
  46. 46
    National Audit Office 2013. Evaluation in Government. National Audit Office, London.
  47. 47
    Noetic Solutions Pty Limited 2012. Report on the Post Incident Analysis of the 2011 Margaret River and Nannup bushfires. State Emergency Management Committee, viewed 17 November 2015,
    [URL]
  48. 48
    NZ Treasury 2011. Economic and Fiscal Impacts of the Canterbury Earthquakes. Viewed 31 March 2016,
    [URL]
  49. 49
    Oklahoma Department of Civil Emergency Management. (n.d.). After Action Report. Oklahoma Department of Civil Emergency Management, Oklahoma.
  50. 50
    Owen, J 2006. Program Evaluation: Forms and Approaches, Allen & Unwin 3rd edition. Crows News.
  51. 51
    Oxfam 2010. From tragedy to recovery: Samoa tsunami response 2009-2010. Oxfam New Zealand, Auckland.
  52. 52
    Plyer, A, Ortiz, E, Horwitz, B and Hobor, G 2013. The New Orleans Index at Eight. The Data Center, New Orleans, viewed 22 November 2015,
    [URL]
  53. 53
    PricewaterhouseCoopers 2010. Effective Disaster Recovery: What Lessons can we learn from Australia’s Black Saturday Bushfires? Viewed 20 November 2015,
    [URL]
  54. 54
    Productivity Commission 2014. Natural Disaster Funding Arrangements-Volume 1. Inquiry Report no.74, Canberra.
  55. 55
    Queensland Farmers Federation 2014. Cyclone Oswald Industry Recovery Initiative viewed 22 November 2015,
    [URL]
  56. 56
    Queensland Flood Commission of Inquiry 2012. Queensland Flood Commission of Inquiry Final Report. Viewed 20 November 2015,
    [URL]
  57. 57
    Queensland Government 2007. The Final Report of the Operation Recovery Task Force. State of Queensland, viewed20 November 2015,
    [URL]
  58. 58
    RAND Labor and Population 2010. The Workforce and Economic Recovery: Effects of Hurricane Katrina. Viewed 20 November 2015,
    [URL]
  59. 59
    Recovery Coordinator 2009. Recovery Coordinator’s Report on the Mid-North and Far-North Coast Flood Recovery May 2009. Viewed 17 November 2015,
    [URL]
  60. 60
    Red Cross n.d. Alberta Floods: One Year Progress Report. Red Cross, viewed 22 April 2015, [URL]
  61. 61
    Regional Australia Institute 2013. From Recovery to Renewal. Viewed 19 November 2015,
    [URL]
  62. 62
    Rich JL, Booth A, Reddy P and Rowlands A 2014. The Step by Step Bushfire Support Service Qualitative Evaluation Report. Centre for Rural and Remote Mental Health, University of Newcastle.
  63. 63
    Robichaud, PR, Elliot, WJ, Wagenbrenner, JW, Lewis, SA, Ashmun, LE, Wohlgemuth, PM and Brown, RE 2010. Continued Evaluation of Post-Fire Recovery and Treatment Effectiveness for Validation of the ERMiT Erosion Model (Paper 41). Viewed 22 November 2015,
    [URL]
  64. 64
    Ryan, R 2014. Presentation of the logic of programs, UTS, unpublished
  65. 65
    Seifert, S and Nieswand, M 2014. What drives intermediate local governments’ spending efficiency: the case of French Départments. Local Government Studies, 40 (5), 766-790.
    [CROSSREF]
  66. 66
    Shkabatur, J 2012. Transparency With(out) Accountability: Open Government in the United States. Yale Law & Policy Review, 31(1), 80-120.
  67. 67
    State Emergency Management Committee 2014. Parkerville Stoneville Mt Helena Bushfire Review. Government of South Australia, viewed 8 December 2015
    [URL]
  68. 68
    State Recovery Committee 2005. Collaboration is Key. Government of South Australia, viewed 24 November 2015,
    [URL]
  69. 69
    Sturgis, LA, Smythe, TC and Tucci, AE 2014. Port Recovery in the Aftermath of Hurricane Sandy: Improving Port Resiliency in the Era of Climate Change. Center for a New American Security, viewed 22 November 2015,
    [URL]
  70. 70
    Tasmanian Bushfire Inquiry 2013. 2013 Tasmanian Bushfires Inquiry: Volume One. Viewed 22 November 2015,
    [URL]
  71. 71
    The Sphere Project 2015. Sphere for Monitoring and Evaluation. The Sphere Project. Geneva
  72. 72
    The Treasury 2013. 2013 Budget: building momentum. Viewed 15 April 2016,
    [URL]
  73. 73
    U.S. Government 2006. An Examination of Federal 9/11 Assistance to New York: Lessons Learned in Preventing Waste, Fraud, Abuse, and Lax Management (20-452). Viewed 22 November 2015,
    [URL]
  74. 74
    UNICEF 2009. Children and the 2004 Indian Ocean Tsunami: Evaluation of UNICEF’s Response in Indonesia (2005-2008). Viewed 22 November 2015,
    [URL]
  75. 75
    VBBRA [Victorian Bushfire Reconstruction and Recovery Authority] 2011. Expert Reference Group Legacy Report. VBRRA, viewed 22 November 2015,
    [URL]
  76. 76
    Victorian Auditor General’s Office 2013. Flood Relief and Recovery. Viewed 22 November 2015,
    [URL]
  77. 77
    Victorian Bushfire Royal Commission 2010. Final Report: Summary. Viewed 20 November 2015,
    [URL]
  78. 78
    Victorian Council of Social Service 2011. Victorian Floods Review. Viewed 20 November 2015,
    [URL]
  79. 79
    Victorian Government Department of Sustainability and Environment 2005. The Recovery Story: The 2003 Alpine Fires. Victorian Government Department of Sustainability and Environment, Melbourne
  80. 80
    Victorian Government of Sustainability and Environment & Parks 2010. 2007 Gippsland Flood/Storm Recovery Program Final Report. State of Victoria Department of Sustainability and Environment & Parks Victoria, Melbourne
  81. 81
    Virginia Horticultural Centre South Australia 2008. Gawler River Flood Recovery Program. Virginia Horticultural Centre South Australia, Virginia
  82. 82
    Warrumbungle Shire Council 2013. Wambelong Redbank Section 44 Bushfire Compilation of Reports. Viewed 20 November 2015,
    [URL]
  83. 83
    WaterSMART 2014. The 2013 Great Alberta Flood: Progress Report on Actions to Mitigate, Manage and Control Flooding. Viewed 20 November 2015,
    [URL]
  84. 84
    Weiss, CH 1995. Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J P Connell, A C Kubisch, L B Schorr, and CH Weiss (ed.s), New approaches to evaluating community initiatives: Concepts, methods and contexts, Vol. 1, Aspen Institute, Washington, DC.
  85. 85
    Yin, RK 1981. The case study crisis: Some answers. Administrative Science Quarterly, 26(1), 58-65.
    [CROSSREF]
  86. 86
    Yin, RK 2003. Case Study Research: Design and Methods 3rd edition. Sage Publications, Thousand Oaks CA.

EXTRA FILES

COMMENTS

  • |