The AIC has identified a significant number of interventions that are supported by evidence of effectiveness in the prevention of the priority crime types currently being targeted by the NSW CPD. It has therefore been possible to draw a number of conclusions based upon the findings presented in this report, both about the effectiveness of different intervention types and the requirements for successful implementation. However, there are important limitations that need to be acknowledged. This section ends with some suggestions for improving the available evidence base to inform local government crime prevention.
Limitations of systematic reviews
There are a number of limitations with the methodology used for this review (and systematic reviews more broadly) that should be acknowledged. The majority of evaluated strategies involved multiple interventions delivered as part of a multifaceted, comprehensive program. Therefore, isolating the intervention or interventions that were most effective is difficult. Similarly, it is difficult to determine the relative contribution of each intervention type to the overall effectiveness of strategies involving multiple interventions. Instead, the research suggests that interventions are most effective when delivered in combination with one another. While there were some exceptions, there was limited evidence that the implementation of a single intervention type would, on its own and in the absence of other strategies, be effective in reducing the priority crime types examined for the purpose of this report.
It is also important to acknowledge that the integration of theory with the experimental method may not entirely overcome some of the methodological limitations that are associated with the absence of a comparison group; specifically threats to internal validity. The quality of evaluations (in terms of the SMS) also varied considerably between the priority crime types. For example, the quality of evaluations of strategies aimed at preventing residential burglary was higher than for malicious damage. This report has been careful not to describe certain interventions as having been ‘proven’ to be effective, particularly as there are a number of intervention types supported by a small number of evaluation studies. Nevertheless, these evaluations have provided valuable lessons both in terms of the process and outcomes from crime prevention interventions and as such, warrant inclusion in the review.
The focus on evaluations for which there is a measure of one of the priority crime types is also likely to have biased the results towards those interventions that deliver short-term outcomes and is likely to have contributed to a greater focus on those interventions that involve the manipulation of situational factors for crime (ie situational crime prevention). Programs targeted at offenders (or people at risk of offending) frequently report on general individual-level outcomes such as self-reported delinquency (for juveniles), arrests and reoffending. They often do not include measures specific to involvement in certain crime types. Where the program has been specifically established to address one of the priority crime types but includes general measures of offending by the recipients or participants in the intervention, the study was included in the review. However, studies were excluded if they did not include a specific measure relating to the relevant crime type(s) and did not have a reduction in one of the priority crime types as an explicit objective. This does not necessarily mean that interventions targeting offenders or individuals at risk of offending would not be effective in preventing the priority crime types examined as part of this review.
Further, since this review was focused on finding evaluations for strategies that addressed the priority crime types, interventions that were identified have then been categorised and general conclusions regarding their effectiveness in addressing the crime types are drawn. This process differs from reviews that have examined the impact of prevention strategies delivered in different settings on crime generally (eg Sherman et al. 2006, 1998) and those that have reviewed the evidence in relation to specific strategies (eg the meta-reviews undertaken on behalf of the Swedish National for Crime Prevention).
There is a risk that following this approach may have led to some studies being overlooked on the basis that they included one measure among many that related to the relevant crime type and were therefore not easily detected as part of the literature search. To overcome this, in instances where a particular intervention has been identified from one study as having impacted (or not) on one of the priority crime type, the research team searched for additional evaluations relating to that intervention (not otherwise detected by the search terms outlined below).
Among those strategies included in the review, few had results that suggested the intervention had been ineffective in reducing the targeted crime problem. It is not surprising that fewer unsuccessful studies were identified compared with those that showed positive results, as it has been recognised that organisations are more likely to release studies that provide favourable results than studies that do not show success. This may serve to overestimate the relative success of strategies examined as part of this review (known as publication bias). Finally, given the short timeframe for this project, it is likely that some evaluation studies have been overlooked, particularly unpublished (or not widely published) and older studies.
The standard of evaluation in crime prevention
These issues aside, it has been demonstrated that the level, quality and strength of the evidence in support of different crime prevention intervention types vary considerably. Some of the prevention approaches described in the introductory sections of this report do not appear to be supported by evidence of effectiveness (or evidence demonstrating that they are not effective). There was strong evidence to suggest that environmental approaches to crime prevention were effective in reducing the priority crime types examined as part of this research. There was less evidence in support of social approaches, even taking into account the methodological limitations described above. For example, there was little research into the effectiveness of diversionary projects targeting young people (particularly as a primary intervention), which are a common strategy employed by local government. This review has also demonstrated that there continues to be limited research into the effectiveness of efforts to modify community-level factors to reduce crime (or involvement in those volume crimes examined as part of the current research project; Homel 2005; Hope 1995; Tonry & Farrington 1995; Welsh & Hoshi 2006).
The age and source of the majority of the evaluations located and reviewed as part of this project is also worth noting. A significant proportion of the studies cited in this report were released more than a decade ago. Further, the majority were from the United States and the United Kingdom, with fewer studies examining the impact of community-based crime prevention in Australia found to meet the criteria for inclusion.
Consistent with previous reviews of situational crime prevention strategies (Eck 2006a), there appeared to be little evidence of displacement of offending to adjacent areas. However, the proportion of studies that examined issues of displacement to surrounding areas (or other forms of displacement) or the diffusion of benefits was relatively low, given the number of studies that examined the impact of situational crime prevention. Similarly, a small number of studies found a long-term positive effect on crime, while others observed that the impact on crime levels appeared to diminish over time. However, relatively few studies monitored the impact of a project beyond 12 months, making it difficult to draw conclusions about the long-term effectiveness of different strategies.
The lack of high-quality evidence from Australia or in relation to common intervention types is possibly due to the fact that most crime prevention programs have, for the most part, placed less emphasis on systematically generating evidence than on the practical application of the available research (Homel 2009a, 2005; Sutton, Cherney & White 2008). The emphasis on short-term and non-recurrent funding has resulted in projects that have been limited in duration and scope, which has impacted on attempts to implement and evaluate interventions that can address the underlying causes of crime. Adherence to the community-based approach to crime prevention has seen much of the responsibility for evaluation devolved to local agencies, including local government and non-government organisations (Anderson & Homel 2005; Anderson & Tresidder 2008; Homel et al. 2007).
Experience from both national and state and territory crime prevention programs suggest that this approach has been largely unsuccessful in terms of generating high-quality evaluations. Numerous initiatives have been implemented that have not been subject to evaluation or ongoing monitoring and where they have been evaluated, the methodological rigour of those evaluations has often been poor (Crawford 1998; English, Cummings & Stratton 2002; Eck 2005; Lipsey et al. 2006; Sherman et al. 2006; Weatherburn 2009, 2004). There are a variety of reasons that this approach has proven problematic, including:
- a lack of clarity and shared understanding of the need for and purpose of evaluation;
- a lack of relevant knowledge, experience and expertise in evaluation among those responsible for overseeing and responsible for conducting evaluations;
- limited training and professional development opportunities to build expertise and capacity to undertake evaluation, a lack of technical support to overcome the challenges posed by a lack of professional experience or expertise and limited access to useful information and resources to assist organisations in undertaking evaluation;
- issues relating to access to useful data and where that data is available, limited expertise in data collection and analysis;
- insufficient resources and in some cases, resistance to allocate resources to evaluation that would otherwise be directed to service delivery;
- application of narrow models of evaluation, often focusing on outcomes at the expense of process or vice versa;
- an emphasis on accounting for financial investment (ie funding acquittal) rather than demonstrating the benefits that have been delivered;
- the length of time allocated to implementing and reporting on projects is often insufficient to observe and measure crime prevention outcomes;
- an inconsistent record of academic interest and engagement in crime prevention evaluation; and
- a failure to commit adequate funding to evaluation (Cameron & Laycock 2002; Crawford 1998; English, Cummings & Straton 2002; Sutton, Cherney & White 2008).
There is considerable scope to improve the way crime prevention programs and projects are evaluated (Cameron & Laycock 2002; Crawford 1998; English, Cummings & Straton 2002; Homel & Morgan 2008; Sutton, Cherney & White 2008).
Improving the quality of evaluation
Comparing the volume of crime prevention activity delivered in Australia and overseas with the number of studies available for inclusion in a review such as this one has highlighted the fact that there is considerable scope for improving the way crime prevention is evaluated. Selecting an appropriate evaluation model requires consideration of the characteristics of a program, the purpose of the evaluation study, the available options and the views of key stakeholders (English et al 2002). Consideration must also be given to the capacity of those conducting the evaluation. There is no single best approach to evaluation; rather there are a variety of different approaches to evaluation, including experimental research designs (including randomised control trials, quasi-experimental designs and pre and post-test comparisons) and realist or theory-driven approaches. There is also a range of different quantitative and qualitative research methods that can be used. In this report, a strong case is made for finding an appropriate balance between evaluation approaches and methods that aim to identify what works in crime prevention through rigorous scientific methods, and those that place greater emphasis on developing a more detailed understanding of good practice and what can be done in what circumstances to prevent crime (Cherney & Sutton 2007; Tilley 2009, 2006).
In addition, where possible, the evaluation of crime prevention strategies should incorporate both process and outcome evaluation. A process evaluation will aim to improve understanding of the activities that are delivered as part of a program. It is also focused on the implementation, operation and management of these activities; assessing whether they were (or are being) delivered as planned and in accordance with the design of the program, determining how well they were delivered (ie to an acceptable standard and the satisfaction of various parties involved) and identifying any factors that may have impacted upon the delivery of these activities. An outcome evaluation is concerned with the overall effectiveness of the program, examining whether the stated objectives have been achieved and determining what outcomes (intended or unintended) have been delivered as a result including the impact of the program on participants, stakeholders and the broader community.
In order to improve the evidence base available to local government around effective crime prevention interventions and the requirements for their implementation, it is important that there are strategies in place to increase both the amount and quality of evaluation research being conducted. This might involve establishing mechanisms to:
- encourage local government and other community-based organisations to undertake or sponsor evaluation work;
- appoint qualified personnel to undertake high-quality evaluation studies on behalf of community-based organisations;
- review evaluation proposals and provide input into evaluation design and methodologies developed by community-based organisations;
- provide guidance and support to local government entrusted with evaluation, in developing the methodology and on an ongoing basis; and
- provide training and develop resources that help to build the capacity of those involved in evaluation and performance measurement (Homel 2009b; Lipsey et al. 2006).
Determining the most effective approach to supporting evaluation work will need to be based on an assessment of the existing capacity and potential needs of those likely to be entrusted with the responsibility for evaluation.
Further research may seek to fill the gaps in the evidence base for local government by targeting specific intervention types for evaluation. Rather than aiming to evaluate all projects, evaluation effort could be targeted at clustered groups of projects (classified according to intervention type, target crime etc) to draw conclusions about effectiveness of specific interventions or projects targeting specific priority areas. Decisions regarding which interventions should be subjected to more rigorous evaluation can be based on an assessment of the potential practical and policy significance of the findings, and of the ability of the intervention to be effectively evaluated (Lipsey et al. 2006). By focusing evaluation on clusters of projects that are identified as being important and/or of interest, the knowledge base on effective crime prevention practice can be developed in a strategic and systematic way. Similar approaches to evaluation have been adopted in New South Wales for graffiti prevention through rapid removal, CPTED and volunteer programs (NSW Department of Justice Attorney General 2009). This approach would be particularly useful for those intervention types that are common in local government crime prevention plans but for which there is little evidence of effectiveness.