The SARP evaluation was conducted in two stages. Stage 1 began in October 2009 and Stage 2 in July 2010. The two-stage process was necessary as the final proposed SARP reforms were introduced in May 2009. As such, a minimum of one year was needed to document changes resulting from these final reforms. Stage 1 was used to identify key indicators for the evaluation and to allow stakeholders providing the data, time to collect the data proposed. It also allowed the researchers the opportunity to ensure the indicator measures selected were feasible in terms of timelines and data extraction. Stage 2 was the key data collection and interview stage. Given the sensitivity of the data collection, particularly in regard to victim/survivor interviews, the proposed evaluation method was submitted to the AIC Human Research Ethics Committee, who approved it.
As discussed earlier, the SARP evaluation sought to address three key questions:
- Have there been improvements in the processes and support for victims of sexual offences as they progress through the criminal justice system?
- Has attrition in sexual offence matters in the ACT’s criminal justice system improved?
- Have there been improvements in the coordination and collaboration among agencies involved in administering SARP?
A mixed-methods approach to the evaluation was adopted: data collected on key measures for each question were supplemented by interviews with both victim/survivors of sexual offences and selected service delivery providers from Wraparound. In planning the evaluation, a classic experimental design evaluation framework was considered unfeasible. Reasons for this include the absence of consistent data across the evaluation, the time parameters for conducting randomised controlled pre-and post-testing, the inability to randomly interview victims/survivors, and the lack of a comparable control site. A realist evaluation framework was consequently adapted and applied.
A realist framework has its advantages and complements other evaluation methods. Instead of focusing principally on what works (such as classic experimental design), realist evaluation looks at the various contexts and mechanisms that are required for a project/program to work (Pawson &Tilley 1997). In other words, the focus is not just on what works but also on how and in what context. Knowing these factors helps agencies determine not only how a project works but also what mechanisms can be altered that might improve service delivery in the future. It also aids greater knowledge transfer among agencies wishing to adopt successful processes from a project (Ekblom 2010) and can help them avoid replicating factors that influenced any negative outcomes (Ekblom 2010).
The realist approach can be applied even if a project has not been completely implemented. Its framework can be used for investigating the processes behind the reforms and what aspects of these appear to have helped (or hindered) furthering SARP in the ACT. The purpose of this evaluation is to give JACS, the SARP Reference Group and Wraparound agencies an insight into how well the reforms have been implemented to date to inform them of any preliminary outcomes. From this assessment, the key stakeholder agencies will be able to reflect on the current practices and identify any service delivery areas that need improving or updating.
Development of indicators
Prior to the AIC evaluation, JACS and Wraparound stakeholders developed a set of indicators for the SARP reforms based on the Responding to sexual assault: The challenge of change report (DPP & AFP 2005). Qualitative and quantitative indicators were identified to measure the success of the reforms according to each objective—for example, ‘number of sexual offence cases reported to police’ and ‘number and type of SARP training sessions delivered’. Indicators were selected on the basis of available data and how feasible these were to access, and on the basis of relevance to the SARP objectives. The preliminary indicator list was updated based on the outcomes of preliminary consultations with key stakeholder agencies, and the AIC then sought feedback on these revised draft indicators from Wraparound agencies. The draft indicator list was again amended based on this feedback and circulated to all Wraparound stakeholders for comment in late 2009. It was then revised and updated in April 2011 to reflect available resources and the feasibility of indicators that were not able to be identified earlier. The final list of indicators is available at Appendix A.
It must be noted that this evaluation has not collected data for each indicator identified, as data for some of the indicators are currently unavailable. Many SARP agencies were in the process of updating or modifying their data collection processes so were unable to provide the relevant information for the period required. However, it was raised during the consultation that these indicators could be incorporated in future program evaluations. Where possible, if quantitative data were not available, qualitative information was sought via the consultations.
Defining a victim/survivor
The term ‘victim/survivor’ is used in this report to describe a person who has been subjected to a sexual offence as defined by Part 3 of the Crimes Act ACT (1900). This Act includes persons who are victims of:
- sexual assault in the first, second and third degrees;
- sexual intercourse without consent;
- sexual intercourse with a young person;
- maintaining a sexual relationship with young person;
- act of indecency in the first, second and third degrees;
- acts of indecency without consent;
- acts of indecency with young people;
- incest and similar offences; and
- abduction (see Appendix C for Part Three 3 of the Crimes Act 1900).
Upon discussion with the DPP, it was determined that all offences under Part 3 of the Act, other than the pornography-related offences, should be included in the parameters of the research. It was noted in initial consultations that victim/survivors of indecent acts can often display as much distress following the offence as victim/survivors of sexual assault, and often require access to the same services.
Inclusion criteria for victims/survivor interviewees
Each service provider that recruited victim/survivors to be interviewed was asked to liaise with the other Wraparound agencies that were assisting the AIC to recruit victim/survivors for interviews. As the timeline and resources for conducting the research were limited, only five victim/survivors in total were interviewed for this study.
The following criteria applied to the victim/survivor selection:
- The participant must be a survivor of a sexual offence as outlined in Part 3 of the Crimes Act ACT (1900) (see Appendix C). (The victim/survivor may have been under the age of consent when the sexual offence occurred, but they must be over the age of consent in order to be interviewed.)
- The participant must be able to understand the purpose and scope of the interview and provide informed consent. For this reason, survivors with intellectual disabilities were excluded.
- Survivors should no longer be ‘in crisis’ and should be able to cope with the demands of the interview with appropriate support mechanisms.
After further consultation with the DPP representative, the criteria for selecting victim/survivors were amended to exclude cases that were pending or had not been finalised in the court. This step was taken as interviewing a victim/survivor prior to the case being finalised could potentially have jeopardised their confidentiality.
CRCC, VSACT and the DPP liaised to determine the most appropriate victims/survivors to approach for the research as well as to avoid an individual being approached by more than one agency. Appropriate protocols protected the privacy of individuals being put forward for the consultations, and each agency reviewed its policies and procedures in relation to what information about each victim/survivor could be shared prior to the meeting. The process began in September 2010 and continued until September 2011. Victim/survivors were approached in October/November 2011 and were granted a three-week ‘cooling-off period’ to carefully consider if they wished to be interviewed.
Interview schedule, method and development
The SARP victim/survivor interview schedule and methodology were modelled on a format developed by a study conducted by the AIC in 2005 for the Office for Women on women’s help-seeking decisions and service responses to sexual offences (see Lievore 2005). This was done because the information sought in the 2005 study was similar to the information that was required for the SARP evaluation. In addition, the 2005 interview schedule was rigorously developed in consultation with an expert in designing questionnaires that request information on sensitive issues, and with feedback from participating sexual offence services (Lievore 2005). The schedule and methodology were also subjected to a rigorous ethics process.
Relevant questions were modified for the SARP evaluation to address the key research questions, and further questions were added as necessary. The interview schedule was then circulated to Wraparound stakeholders for comment. The final draft interview schedule, incorporating the stakeholder comment, was submitted to the AIC Research Human Ethics Committee for consideration.
The interviews with victim/survivors were conducted following a semi-structured format. Victim/survivors had the option to cease the interview at any time, and counselling services were available to them at any stage during and after the interview.
Sexual offence service delivery provider (SDP) interviews
Interviews with the various sexual offence SDPs in the ACT comprised a key component of this study. However, due to time and resource limitations it was not feasible to interview all agencies involved in the SARP reforms. In the end, five interviews were conducted with representatives of the following key Wraparound agencies: ACT Policing SACAT team, CRCC, VSACT, and the DPP. The ACT Courts (although not formally part of Wraparound) were also interviewed. These interviews were conducted using a semi-structured format and were based on the key objectives of the reforms. Questions for each agency varied slightly to reflect its role and level of involvement in Wraparound. Only one agency, JACS, was not interviewed face-to-face; at its own request it was sent out a modified questionnaire to complete.
Limitations of the study
The current SARP evaluation was very limited in both resources and time, so its scope was developed to reflect these restrictions. A key limitation in this evaluation is that the data available were not able to be compared across agencies. Further limitations are listed below.
Different units of measurement used
When different units of measurement are used in various datasets, it is difficult to determine whether what is being analysed is comparing like with like, This can reduce confidence in the findings—particularly in the case of data collected from the different agencies participating in the SARP reforms. Wraparound data are victim-focused, in contrast to data collected from the ACT Courts and the DPP, which are offender-focused. This is problematic because an offender can have multiple victims and/or multiple offences and may therefore be counted numerous times in captured data. Similarly, victims can have multiple offenders. This limitation was identified in a previous study tracking sexual offenders in the ACT criminal justice system (see Borzycki 2007).
Nature of available datasets
Although collation of the indicator data has been quite consistent in the last 12 months, available indicator data from before the 2009 legislative changes are limited across all agencies. Most of the indicator data identified as appropriate for the evaluation have not been collected in their present format for the period of data collection required. Indeed, as the SARP reforms have been implemented only recently, it is difficult at this stage to draw many conclusions from quantitative data. As a result, this preliminary evaluation has relied heavily on qualitative data collected from key stakeholder agencies and the small number of victim/survivors of sexual offences.
Delays in criminal justice system proceedings
Although it was suggested that many of the causes of delays in sexual offence cases pre SARP have been minimised by the new legislative changes (DPP 2009), delays still exist, and some sexual offence cases in the criminal justice system can take years to be resolved. As few complainants have progressed through the criminal justice system since the reforms began, it is difficult to look at changes over time. This evaluation, as mentioned, has therefore relied heavily on qualitative data from stakeholders and victim/survivors. Future evaluations will be better placed to make more conclusive comments about the impacts of the SARP reforms.
Incremental implementation of SARP reforms
The suite of SARP reforms proposed in the DPP and AFP (2005) report has been implemented incrementally since 2007 and, of these, the most recent legislative changes occurred in May 2009. As there is no fixed date on which the legislative changes came into effect, no comparison can be made of the system pre and post reforms. In any case, the effect of the SARP reforms may have predated the data collected for this evaluation, but data for an earlier period would have been unreliable so was not collected.
Applicability to specific victim/survivors
The study excluded looking specifically at the effects of SARP on victim/survivors from CALD backgrounds, of Aboriginal and/or Torres Strait Islander descent, and/or with mental illness, intellectual disability or other disabilities, primarily because of the additional resources needed to cater for these groups in the interview process. However, victim/survivors from these backgrounds were not actively excluded from the interview process if they were able to satisfy the interview inclusion criteria.
The small ACT population and small number of victim/survivors who report sexual offences mean the sample of people identifying with one or more of these groups is very low. They are nonetheless recognised as a priority for future research into criminal justice responses to sexual offences, as it has been recognised that women from Aboriginal and/or Torres Strait Islander and CALD communities and/or those with a mental health issue or intellectual disability may be particularly at risk of sexual violence.
A national survey on violence against women conducted by Mouzos and Makkai (cited in Taylor & Putt 2007) found that sexual violence against Aboriginal and/or Torres Strait Islander women was three times more common than against non-Aboriginal and/or Torres Strait Islander women. It was also found that CALD women were as much at risk of sexual violence as non-CALD women. In addition, a recent report on violence against women with disabilities highlighted that they often suffer higher levels of domestic/family violence and sexual offences than women without a disability, and that support services catering to these women are limited or not available (see Frohmader (2011) for an overview of these factors in more detail).
Bias in stakeholder and victim/survivor interviews
Stakeholders interviewed for the evaluation have a significant investment in the success of the SARP reforms. Many of the Wraparound agencies and SARP Reference Group members received funding to implement their part of the reforms. As such, there is a potential risk for stakeholders interviewed to describe the reforms in a favourable light. Key stakeholders interviewed for this research from Wraparound were self-selected. In addition, focus group attendees who were not part of the Wraparound/SARP Reference Group (eg police investigators) were selected by their agency’s Wraparound contact officer. Despite these risks to objectivity, the stakeholder interviews were an appropriate method to obtain information on the SARP reforms, as there are no others in the community who would be able to comment on these reform processes and their success. In addition, victim/survivor interviews have been used to balance the perception of stakeholder agencies.
Limited pool of suitable victim/survivors to interview
Many sexual offence cases are settled via a guilty plea (see Table 5 above), limiting the number of sexual offence cases defended in the courts since introduction of the SARP reforms. This substantially lowered the number of victim/survivors who could be recruited to comment on the support services provided and on aspects of the reforms throughout the whole process.
The small number of interviews conducted for this preliminary evaluation of the SARP reforms (n=5) cannot be considered representative of all sexual offence victim/survivors in the ACT. Nonetheless, the interview data that emerge provide an important insight into the experiences of a small number of victim/survivors, and a critical complement to the views of service delivery providers and other stakeholders.