A framework for measuring the performance of drug law enforcement

Foreword | It has been estimated that Australia expends between $1.3 and $2 billion annually on drug law enforcement activity. Such a high level of effort and commitment means that it is vitally important to have good systems in place to measure the efficiency and effectiveness of law enforcement action to reduce drug problems in the Australian community. However, Australia's drug law enforcement activities are undertaken in conjunction with efforts to reduce demand and levels of drug related harm, coordinated through the National Drug Strategy. Accordingly, identifying the specific contribution of drug law enforcement to the achievement of Australia's goals of reducing drug related harm, demand and supply is not a simple task. As part of an attempt to answer this question, the National Drug Law Enforcement Research Fund commissioned the Australian Institute of Criminology to develop a model performance measurement framework for drug law enforcement (DLE) in Australia. The model that was developed encompassed both a core performance measurement framework built around four high level outcomes and a process for adapting the framework to accommodate the specific needs of the range of DLE agencies operating in different settings around Australia. The project has also demonstrated the value of applying modern performance measurement processes to complex multi-agency collaborative activities to help clarify and explain how shared goals and outcomes can be achieved and assessed.

Toni Makkai
Director

Drug law enforcement (DLE) agencies in Australia and overseas have used drug seizure and arrest data to measure the effectiveness of their work performance for many years. Such measures are simple, visible and well-understood measures of law enforcement effort. However, they are frequently ambiguous measures of law enforcement performances as they essentially demonstrate the extent to which law enforcement agencies engage in certain types of activities rather than demonstrating the broader impacts of law enforcement work. For example, they reveal little about the impact of DLE activity in contributing to the achievement of important broad outcomes, such as making communities feel safer and more secure. Nevertheless, it is these broader impacts that are often described by DLE personnel as being important drivers and rewards for persisting with the DLE effort. This paper provides a description of a model performance measurement framework for DLE in Australia that was developed in an attempted to redress some of the evident limitations of the current systems available in Australia. The framework's development formed a major component of a research project undertaken by the Australian Institute of Criminology on behalf of Australia's National Drug Enforcement Research Fund to provide better, appropriate and meaningful information that could inform the most effective interventions within DLE in Australia.

This paper provides a description of a model performance measurement framework for DLE in Australia that was developed in an attempt to redress some of the evident limitations of the current systems available in Australia. The framework's development formed a major component of a research project undertaken by the Australian Institute of Criminology (AIC) on behalf of Australia's National Drug Law Enforcement Research Fund to provide better, appropriate and meaningful information that could inform the most effective interventions within DLE in Australia.

DLE agencies around the world have long grappled with the limitations of traditional DLE performance measurement systems. In a review of drug data and research available in the United States of America, Manski, Pepper and Petrie (2001) found that existing sources of information and performance measurement systems built around this information were inadequate to assess the extent of the illicit drug trade in that country. Similar difficulties have arisen in the United Kingdom (Newburn & Elliott 1998). Perhaps the most significant Australian research to date on this subject was conducted by Sutton and James (1996). They found that no DLE agency was satisfied with the performance measures available to them, but that there had been little (if any) attempt to tackle the problem internally. The past decade has seen only a few Australian attempts to develop more robust DLE performance measurement systems (NSW Police 2002; Weatherburn 2000).

The enforcement of laws relating to the production, importation and distribution of illicit drugs is a major investment for the Australian community. In the early 1990s, Marks (1992) estimated that around $320 million was expended on DLE in Australia each year. This included costs from DLE work conducted by the Australian Customs Service (Customs), the Australian Federal Police (AFP) and the various state and territory police drug squads and units, as well as costs from the judicial and correctional systems. Recent estimates of the annual direct costs of DLE in Australia vary from $1.4bn (Collins & Lapsley 2002) to $1.96bn (Mayhew 2003), each of which is probably an underestimate. Moore (2005) estimated total government spending to counter illicit drugs at $1.3bn for 2002-03. Of this, law enforcement represented 42 percent and interdiction 14 percent, together comprising the majority of spending. Prevention represented 23 percent and treatment 17 percent.

Australia has a complex array of DLE agencies spread between the national and state/territory levels. Unlike many other countries, Australia does not have any law enforcement agencies exclusively dedicated to drug law enforcement. While many law enforcement agencies have specific units working exclusively on illicit drug issues, these responsibilities are handled as part of their broader law enforcement functions. This is the result of a number of factors including:

• the federal system and associated legislative arrangements
• variations in the production and distribution systems for different drug types
• local and regional variations in drug consumption and availability
• organisational priorities for police.

Moreover, the management of DLE in Australia is not static. At any particular point in time a specific jurisdiction may be reviewing or reorganising the way DLE activities are planned and delivered. However, it is important to understand that DLE occurs at many levels in Australia, from general duty policing to drug unit and command levels, through to state and federal levels, as well as across jurisdictions. Each level and jurisdiction differs in terms of whether there is a dedicated or generalist organisational structure and whether this is primarily centralised or regionalised in nature. They also differ according to preferred operational approaches, the number of specialised personnel employed, legislation, agency resources and the character of the different markets for illicit drugs themselves.

The basic relationship between each DLE agency is illustrated in Figure 1. National agencies are generally distinguished from state/territory agencies in terms of their focus on external or border protection goals rather than domestic production, distribution and consumption. However, they frequently work together on joint operations as well as in partnership with non-DLE agencies.

Generally, the state/territory DLE management model is one where there is a centralised crime agency within a given police service that undertakes specialist DLE operations at jurisdictional and cross-jurisdictional levels. Other specialist investigative agencies that also exist in a number of jurisdictions (for example, the NSW Crime Commission) frequently focus on DLE work, especially when it involves high level organised crime. A mixture of specialist drug units and/or general duty police units at particular local police commands manages local level state/territory DLE. How this is organised is largely dependent on the nature of illicit drug market activity in specific areas. National level DLE is managed principally by Customs and the AFP, together with the Australian Crime Commission, and is focused on drug interdiction, particularly at Australia's border and internationally.

While the description of Australian DLE provided here suggests that each DLE management level is discrete, in practice there is often crossover between the various levels, so that collaborations and partnerships are formed as the need arises. For example, a centralised state/territory crime agency and the AFP may pool investigative work on an operation in a particular area. Similarly, Customs and the AFP routinely collaborate with each other as well as working together with other agencies overseas on DLE operations conducted outside Australia's borders. This provides the level of flexibility needed for dealing with a constantly changing drug market.

While it remains true that Australian DLE action is primarily directed at impacting on reducing the supply of drugs into and within the country, it became clear during the first stage of the project that developing a set of performance measures that merely sought to assess supply reduction outcomes would be inadequate. In practice, DLE policy in Australia reflects the tripartite goals of the National Drug Strategy - reductions in harm, supply and demand. This was very apparent in the detailed interviews undertaken with around 100 key law enforcement officers across the country currently involved in DLE activity. An examination of the variety of strategic plans used by national, state and territory DLE agencies during the project further disclosed that expectations for the range of impacts and effectiveness of DLE actions extended well beyond supply reduction alone and into goals associated with improving community wellbeing (i.e. public amenity) as well as the health and wellbeing of drug-involved individuals.

In developing and testing a viable performance measurement framework for DLE work in Australia, the project accepted that the range of measures that would need to be included would need to address the full scope of the anticipated impacts - that is, across the areas of drug and drug-related crime, health and public amenity. However, the project also needed to consider four important challenges.

First, much of the data available for measuring the performance of DLE action in achieving goals such as improvements in public health and public amenity was either not readily available to law enforcement agencies or was poorly developed. Additionally, the measures collected by other agencies tended to reflect the needs of those agencies and were not always easily adapted to other uses.

Second, there is a need to avoid having too many performance measures, as well as developing too many measures from scratch. The practical basis for using a small number of individual measures is that it becomes easier to properly attribute cause and effect in terms of actions taken and outcomes achieved.

Performance measurement, when used as a tool for performance improvement, needs to be based on familiar measures - that is, measures that are understood and accepted by those engaged in the work being examined. Accordingly, the project sought to identify key measures that could be promoted and developed rather than attempt to introduce entirely new measures. Therefore the framework for the set of recommended performance measures is linked to each identified high level outcome and an existing data source. It is recognised that not all data sources identified for use in the model would be available in all settings and the framework represents a model instrument that needs to be adapted to suit different organisations within different settings.

The final major challenge for the project was developing a performance measurement framework that was dynamic. The high level outcomes for the framework are all directional - that is, they point to either improvements or reductions in drug-related conditions. This means that the context in which the performance measurement system will operate is a dynamic one where change, hopefully in the desired direction, will constantly occur. Significant elements of the wider environment will always be outside the direct control of program managers and need to be accounted for in any performance measurement system. This means that a performance measurement framework needs to be regularly reviewed and updated.

The framework was a product of a number of different stages of development, including:

• project meetings, workshops and over 100 interviews held with personnel in DLE agencies in every jurisdiction throughout the various stages of the project
• discussions held during the project's regular advisory committee meetings
• a detailed examination of the available DLE performance measurement literature (both published and unpublished).

However, much of the framework's development occurred through implementation trials undertaken as part of the project in two field sites: a site with a national focus (Customs), and a site with a local focus (NSW Police). An in-depth examination of the appropriateness and feasibility of each of the framework's measures occurred at both of these sites. Key elements of the framework were further developed and tested using field trials in two NSW Police Local Area Commands. These trials focused on the development of a supplementary tool built on the experience and insights to emerge from the progressive implementation of the AIC's Drug Use Monitoring in Australia (DUMA) program (see Mouzos, Smith & Hind 2006). This tool involved the enhancement of a standard offender debriefing process currently applied to all arrestees in most Australian jurisdictions for the purpose of gathering local crime intelligence to include questions on illicit drug market activities.

One of the clear messages from the trial exercises in both Customs and the NSW Police was that without strong executive commitment to the implementation of the performance measurement framework, the system would flounder. However, the measures employed also needed to be meaningful and relevant to those working at all levels of the DLE process - a particular challenge given the different foci of DLE at the national, jurisdictional and local levels. However, the basic principles and the basic steps were clear:

• develop multiple high-level outcomes
• identify adequate measures
• develop methods for dealing with outcome time lag (that is, some initiatives will take longer than others to achieve their goals)
• identify tools for attributing outcomes to interventions.

In working with complex programs with multiple strategic and operational players employing a mixture of interventions, it is essential to establish a clearly defined set of high level outcomes that can be seen to be associated with the program as a whole. A lack of clarity around high level outcomes renders performance measurement meaningless. Good performance measurement relies on a foundation of consensus about objectives. Ambiguity and conflict in goals and outcomes is normal, not unusual. Their interaction must simply be accounted for.

What emerged from the project was that there was a great deal of clarity of the objectives for DLE across the various levels of DLE in Australia. This was strengthened by the strong coincidence of DLE goals with those of Australia's National Drug Strategy. As a consequence, the project was able to focus on the last three steps in the process for developing DLE performance measures.

Performance measures developed for the model framework (Table 1) underpin four high level outcome areas identified by DLE personnel during the project as key outcomes of DLE effort. In turn, and as highlighted above, these outcomes support Australia's National Drug Strategy goals to limit the supply of and demand for illicit drugs, while also minimising community harms.

The model framework's four high level outcomes are:

• reducing drug crime and drug-related crime
• reducing organised crime
• improving public health
• improving public amenity.

The first outcome, reducing drug crime and drug-related crime, includes measures directed at addressing specific drug crimes (for instance, the importation, supply and distribution of illicit drugs), measures for assessing drug market dynamics, as well as a measure of the crime most reliably associated with illicit drug use (robbery). For example, to assess changes in drug markets, the framework includes a series of measures related to drug price, purity and availability, as well as measures concerned with drug trafficking practices. The distinction in the framework between drug crime and drug-related crime is deliberate and made because the latter is often used loosely to describe both types of crime, when in fact their aetiologies are quite different.

The second outcome, reducing organised crime, includes measures specifically directed at addressing high level drug crime. It is distinguished from the framework's first outcome because of the other crimes that usually go hand-in-hand with organised criminal groups that traffic illicit drugs (such as money laundering, extortion, corruption of public officers) and that have serious and far reaching impacts on the community's safety and welfare. Measures for this outcome focus on elements concerned with trafficking. As can be seen in the framework, a number of the same measures are also incorporated under the first outcome as they may also be applied at a more local DLE level.

The third outcome, improving public health, includes a range of measures for gauging the impact of illicit drugs on the community's health. For example, trends in illicit drug-related deaths and morbidity and the health services underpinning these are included in the framework. The fourth outcome, improving public amenity, incorporates a small number of measures of community safety and wellbeing.

While separated for the purposes of reporting, in practice the four high level outcomes are interrelated, not discrete. The nature of this interrelationship is complex and varied but how it can operate is illustrated in Figure 2.

For example, activities directed at reducing aggregate drug consumption and expenditure (A) are likely to impact on all four high level outcomes. On the other hand, measures specifically targeting crime problems associated with illicit drugs, such as money laundering and extortion (E), are likely to have the most impact on reducing organised crime, and so on. However, because of these linkages, it is likely that activities undertaken by DLE agencies to reduce drug crime and drug-related crime may also contribute to improving public health and/or public amenity. In practice this can be seen in moves both in Australia and overseas to link enforcement action to treatment provision. This aims to ensure that any disruption or depletion of a drug market is sustained by providing treatment and support to drug users if and when there is a decrease in the availability of drugs (Scotland. Effective Interventions Unit 2004). A further related issue is that each outcome is constrained by every other outcome. As such, activities directed towards one outcome should not adversely impact on the other outcomes. For instance, police do not generally target clients in the vicinity of drug clinics and limit their attendance at overdoses as the priority in these cases is to minimise drug-related harm.

Conclusion

The project from which the framework was developed demonstrated that it was possible to apply the principles and tools of the modern performance measurement field to the development of a potentially viable performance measurement framework for DLE. Furthermore, the framework that has been developed is sufficiently flexible to address the needs of national DLE agencies with a brief for border protection and offshore operations as well as state and territory DLE agencies concerned with a mixture of responsibilities ranging from organised crime suppression to street-level dealing and associated local crime problems. However, because the framework has deliberately been designed to be flexible, it should not be seen as a one size fits all prescription for all DLE agencies. The framework is a model and starting point for the development of appropriate performance measures for specific agencies with specific briefs in different settings.

The framework's major strength is that it incorporates a range of measures that are, among other things, clear in their purpose, useful in gauging the effectiveness of DLE policies and strategies, reliable and easy to interpret. They are also easy to adapt to different settings and, importantly, are aligned with Australia's wider drugs policy environment, particularly the National Drug Strategy. However, the most important aspect of the framework is that the measures, especially those relating to drug crime and drug-related crime, mark an important shift in emphasis away from viewing drug seizure and arrest measures as the chief means for assessing DLE work performance. Instead, this new suite of measures will permit the monitoring of changes in the international and domestic drug markets, providing a more powerful tool for assessing the real impact of DLE on illicit drug market activity. What is now required is a commitment by a variety of DLE agencies to adopting the framework and to commence implementing it. In this way the real practical value of the model and the processes developed will be properly tested in a range of settings beyond where it was trialled. This will be the only real way of knowing how well it works in practice.

Acknowledgements

This project was funded by the National Drug Law Enforcement Fund. The full report, Developing and implementing a performance measurement framework for drug law enforcement in Australia is available at http://www.aic.gov.au/publications/other/2006-ndlerfmono18.html

References

• Collins DJ & Lapsley HM 2002. Counting the cost: estimates of the social costs of drug abuse in Australia in 1998-99. National Drug Strategy monograph series no. 49. Canberra: Department of Health and Ageing
• Manski C, Pepper JV & Petrie CV (eds) 2001. Informing America's policy on illegal drugs: what we don't know keeps hurting us. Washington DC: National Academy Press
• Marks R 1992. The costs of Australian drug policy. Journal of drug issues 22: 535-548
• Mayhew P 2003. Counting the costs of crime in Australia. Trends & issues in crime and criminal justice no. 247
• Moore TJ 2005. What is Australia's "drug budget"? The policy mix of illicit drug-related government spending in Australia. Drug Policy Modelling Project Monograph Series. Fitzroy: Turning Point Alcohol and Drug Centre
• Mouzos J, Smith L & Hind N 2006. Drug use monitoring in Australia: 2005 annual report on drug use among police detainees. Research and public policy series no. 70. Canberra: Australian Institute of Criminology
• Newburn T & Elliott J 1998. Police anti-drugs strategies: tackling drugs together three years on. Crime detection and prevention series 89. London: Home Office
• NSW Police 2002. NSW illicit drug law enforcement indicators. Sydney: NSW Police
• Scotland. Effective Interventions Unit 2004. Reducing the impact of local drug markets: a research review. Edinburgh: Scottish Executive
• Sutton A & James S 1996. Evaluation of Australian drug anti-trafficking law enforcement. Report series no. 128. Adelaide: National Police Research Unit
• Weatherburn D 2000. Performance indicators for drug law enforcement. Contemporary issues in crime and justice no. 48. Sydney: Bureau of Crime Statistics and Research
• Willis K, Homel P & Gray K 2006. Developing and implementing a performance measurement framework for drug law enforcement in Australia. Monograph series no. 18. Adelaide: NDLERF

About the authors

Peter Homel is a Senior Research Analyst and Manager, Crime Reduction and Review Program at the Australian Institute of Criminology. He is also an Adjunct Professor at Griffith University's Key Centre for Ethics Law Justice and Governance.

Dr Katie Willis is a research analyst in the Crime Reduction and Review Program of the Australian Institute of Criminology.