Ethics & Governance Metrics

Delays in the clinical trial approval and enrolment process are often cited as contributing to a decline in Australia’s clinical trial competitiveness; however, this perception is difficult to verify without reliable data. NSW Health has responded by implementing a number of reform initiatives which are designed to contribute to a significant and sustained improvement in NSW’s ability to initiate and deliver health and medical research. As part of this process, metrics will be used to measure the success of the reform agenda and to help identify areas where there may be opportunities for process improvement.

The metrics support the following objectives:

• to increase the number of commercial clinical trials conducted in NSW
• to reduce the time taken to obtain research approvals
• to increase the proportion of trials in NSW that enrol the agreed number of participants within the agreed timeframe
• to continuously improve the quality of Human Research Ethics Committee (HREC) review and to reduce the time taken for ethical approval
• to continuously improve the quality of Site Specific Assessment (SSA) review and to reduce the time taken for site authorisation
• to improve the quality and compliance of clinical trials conducted in NSW.

NSW Health will continue to collect institutional data for six metrics to capture a benchmark position and gauge progress towards meeting its objective to position NSW as a preferred destination to conduct commercial clinical trials. These metrics are outlined below.

Metrics for commercial clinical trials greater than low risk

1. The number of new commercial trials in NSW in the reporting period.
2. NSW Human Research Ethics Committee Submission Closing Date to first NSW participant enrolled.
3. First participant enrolled to a commercial clinical trial by the site within 40 calendar days of site authorisation (2017-18 Service Agreement improvement measure – MS5302) (available for NSW Health staff only).
4. Actual participants enrolled to a commercial clinical trial as a proportion of those initially agreed to be enrolled per the Clinical Trial Research Agreement minimum target (%) (2017-18 Service Agreement improvement measure – MM5301) (available for NSW Health staff only).
5. Sites reaching or exceeding their agreed enrolment target as per contract.

Further metrics for all health and medical research greater than low risk

6. Ethics applications involving more than low risk to participants approved by the reviewing HREC within 60 calendar days (%) (2017-18 Service Agreement Key Performance Indicator -KS5301) (available for NSW Health staff only).
7. Site specific applications involving more than low risk to participants authorised within 30 calendar days (%) (2017-18 Service Agreement Key Performance Indicator -KS5302) (available for NSW Health staff only).

The 2016-17 Improvement Measure 5, “Progress reports on all authorised research projects received at least annually and at study close”, will not be collected in 2017-18. We hope that the systems established to track site-level progress reporting in 2016-17 will benefit the monitoring of active research projects into the future. Data collection and analysis for progress reporting may resume once REGIS is fully operational.

Non-reportable quality and transparency metrics for internally-sponsored clinical trials, for optional use within an institution

NSW Health also recommends that institutions sponsoring clinical trials collect data on an additional four metrics in accordance with the Alltrials campaign. These optional metrics may be collected and used by institutions to further research quality and transparency objectives in support of statewide initiatives.

8. Internally sponsored clinical trials in NSW that are registered on a publicly accessible database before the first participant is enrolled.
9. Protocols for internally sponsored clinical trials in NSW that comply with the 33-item SPIRIT checklist.
10. Reports of trial summary results (Summary Reports) for internally sponsored clinical trials in NSW posted publically within 12 months of study completion.
11. Full reports of trial results (Final Study Reports) for internally sponsored clinical trials in NSW that comply with the 25-item CONSORT checklist or, where applicable, CONSORT Extensions.

Most data will continue to be reported by institutions to NSW Health through the NSW Australian Research Ethics Database (AU RED). Further information about the collection of metrics in REGIS will be provided in advance of the REGIS launch. OHMR has developed a Research Governance Metrics Data Collection and Reporting Manual for all participating public health organisations, which outlines the reporting parameters, AU RED data entry fields, requirements, definitions and other functionality.

NSW Health is updating the Research Ethics and Governance Metrics Data Collection and Reporting Manual in accordance with the Research Ethics and Governance Information System (REGIS). The manual will outline the metrics to be reported via specific REGIS fields, including metrics calculation parameters and data entry deadlines.

These frequently asked questions address NSW Health’s collection of health and medical research governance metrics data from NSW local health districts, specialty networks and NSW Ambulance.

These FAQs were written when NSW Health used the Australian Research Ethics Database (AU RED) for research ethics and governance processes, and thus refer to that database and its related measures. These will be updated to reflect the Research Ethics and Governance Information System (REGIS) in mid-2018.

NSW Health will periodically update the frequently asked questions for this metrics section for participating institutions. If you have any additional questions about the metrics or data collection, please get in touch with the Research Ethics and Governance team.

What Metrics will be collected in 2017-18?

The following four metrics have been incorporated into the 2017-18 local health district and network Service Agreements issued to the respective districts and networks in late June 2017:

• ethics applications involving more than low risk to participants approved by the reviewing HREC within 60 calendar days (%) (Metric 3)
• site specific applications involving more than low risk to participants authorised within 30 calendar days (%) (Metric 4).

Monitoring measures

• first participant enrolled to a commercial clinical trial by the site within 40 calendar days of site authorisation (Metric 5)
• actual participants enrolled to a commercial clinical trial project as a proportion of those initially agreed to be enrolled per the Clinical Trial Research Agreement (CTRA) minimum target (%) (Metric 6).

NSW Health will also continue to collect and analyse data for the following metrics in 2017-18:

• the number of new commercial trials in NSW (Metric 1)
• NSW HREC submission closing date to first NSW participant enrolled (Metric 2).

What metrics will NOT be collected in 2017-18?

2016-17 Monitoring Measure 5, “Progress reports on all authorised research projects received at least annually and at study close” (Metric 7), will not be collected in 2017-18. We hope that the systems established to track site-level progress reporting in 2016-17 will benefit the monitoring of active research projects into the future. Data collection and analysis for progress reporting may resume once REGIS is fully operational.

Why do the metrics calculations exclude low and negligible risk research?

Initially, data collection for metrics will focus primarily on clinical trials, in conjunction with NSW Health’s objective to position NSW as a preferred destination for clinical trials. Although low and negligible risk research comprises a large proportion of projects reviewed within many NSW public health organisations, the majority of clinical trials reviewed are greater than low risk.

NSW Health will consider the collection of additional data in the future, including those for low and negligible risk research projects.

Why are metrics 2 and 5, which measure time to first clinical trial participant enrolment, calculated from site authorisation date rather than from site activation date?

The use of the NSW Australian Research Ethics Database (AU RED) to collect metrics-related data provides NSW Health and institutions with an historical picture of timelines from SSA application submission to site authorisation, based on existing fields. Although a new field for site activation may be added in the future, all institutions currently use – and are familiar with – the site authorisation date field.

By choosing to calculate from date of site authorisation rather than site activation, NSW Health was conscious of keeping to a minimum the number of new data elements that sites would be required to collect. Additionally, by measuring from site authorisation date, one of the intents is to encourage institutions to work more closely with clinical trial sponsors to facilitate timely study start-up.

NSW Health recognises that sponsors sometimes delay site activation (e.g. “I.P. green light”) for reasons beyond the site’s control. Two additional data sets will therefore be reviewed when analysing this metric:

1. New AU RED fields on the SSA Post Approval Tab allow institutions to enter a valid reason for a delay in first participant enrolment. One available drop-down option is “significant sponsor-related delay in Site Activation,” and a comments field is provided.
2. Institutions are encouraged to change Study State on the SSA Post Approval Tab to “started” to record (and provide for analysis) the date of site activation.

To ensure these data are available, institutions are encouraged to collect the date of site activation and any valid reasons for enrolment delay from investigators of – at least – all commercially-sponsored clinical trials greater than low risk, and to enter these data into AU RED by the required data entry deadline.

For metrics 2 and 5, (time to first clinical trial participant enrolment), why is date of participant enrolment calculated from “date first participant consented and screened, with eligibility verified,” rather than from the date the first participant was recruited, consented and/or pre-screened or screened?

Metrics that measure study start-up duration from ethics application submission to the first participant enrolled drive improvement in clinical trial delivery. While institutions sponsoring and/or hosting clinical trials are encouraged to collect data on multiple study start-up milestones, NSW Health initially selected three to reduce the initial burden of data collection: date of site authorisation; date of site activation (via Study State); and, date of first participant enrolment.

Enrolment was selected rather than recruitment because its definition remains consistent regardless of screening and pre-screening requirements, and because recruited participants do not always enrol in the study. Furthermore, commercial clinical trial sponsors measure success based on the number of enrolled participants that – ideally – complete all study visits.

As part of the metrics program evaluation, NSW Health will request feedback to determine whether additional fields, such as date first participant recruited and/or consented, should be added to AU RED.

NSW Health recognises that, especially at the beginning of the exercise, there will be a number of reasons why the 40-day time to first participant enrolment benchmark is not met. Reporting institutions are therefore encouraged to provide an explanation in the “valid reason for not enrolling within 40 days” fields. Institutions may also choose to note in the comments field if the first participant was nevertheless consented within 40 days from site authorisation.

Based on the new clock start definition for SSA applications, will NSW Health calculate 30 days for monitoring measure 2 (Site specific applications involving more than low risk to participants authorised within 30 calendar days), from date SSA application received, or from date of clock start/application deemed valid?

NSW Health will calculate 30 days to site authorisation based on SSA application clock start date, which is the date on which an SSA application has been deemed valid by the RGO. This is the date the SSA has been received by the RGO and is confirmed to contain the following signatures: all investigators who will conduct research at the site; and the head of department (or divisional director or other authority) of the Principal Investigator at the site.

A research office’s procedures and materials should encourage investigators to submit an SSA application only when it meets the definition of valid. Furthermore, the office is encouraged to determine an application’s validity promptly upon receipt.

In addition to clock start date, NSW Health will collect and analyse the date the SSA application was received, which is the date the RGO or designee receives an SSA application from a researcher regardless of whether or not it is complete and/or deemed valid. Ideally, the date an SSA application is received should be very close to the date an SSA application is deemed valid.

How should an SSA application be managed in AU RED if it is deemed valid per the new definition, but does not contain the required documents that complete an application?

Per the Research Governance Metrics Data Collection and Reporting Manual, the SSA application Clock Start Date should reflect the date on which the SSA application is deemed valid by the RGO. This will be the date that the SSA has been confirmed to contain the following signatures: all investigators who will conduct research at the site; and, the head of department (or divisional director or other authority) of the Principal Investigator at the site. In other words, if the application contains these signatures, the SSA application clock should be set to reflect the date the application was reviewed for these signatures. Ideally, given the minimum requirements for a valid application, the date the application was determined to be valid/invalid is the same date on which it was received by the RGO.

If the application contains the signatures but does not contain the documents required for a complete SSA application, the SSA application clock should then be stopped to reflect the date that the written ”Request for Further Information” is provided to the investigator.

In all cases, the AU RED application receipt date should reflect the date the RGO or designee receives an SSA application from a researcher, whether or not it is complete and/or deemed valid. If the application does not contain the signatures above, the RGO should still record the application receipt date in AU RED, but should not start the SSA clock until the signatures are obtained (that is, when a valid application is received).

Why are the metrics data collection parameters for each reporting period based on the last chronological data point to occur within a sequence, rather than the first? For example, for metric 4 (date application received to site authorisation), the parameters include all studies authorised within the reporting period, rather than all applications received within the reporting period.

NSW Health aims to maintain consistency in the analysis of data for all metrics for the coming years; therefore, each metric (excluding metric 5) will be analysed based on studies wherein the last chronological data point to occur within a sequence falls within the reporting period. This allows NSW to finalise and report on a complete data set as soon as possible after the end of the data collection period.

As an example, a complete data set for metric 4 (Date Valid SSA Application Received by the RGO to Site Authorisation) will be available for all studies authorised within a reporting period shortly after the data entry deadline (which may occur soon after the end of the reporting period). If the data was collected based on date of application receipt, any applications received within the period would have to wait until they were complete until the report was able to be generated.

Metric 5 (Sites enrolling their first participant within 40 calendar days of Site Specific Authorisation) is based on date of site authorisation, rather than the last chronological data point, to ensure consistency between studies represented in metrics 1 and 4.

NSW Health recognises that this approach is different than that initially proposed and communicated (e.g. in the distributed Information Sheet). Some funding from the year 2 grant allocation may be used to support additional data collection and entry efforts, if additional resources are needed after 1 July 2016.

These frequently asked questions address NSW Health’s collection of health and medical research governance metrics data from NSW local health districts, specialty networks and NSW Ambulance.

These FAQs were written when NSW Health used the Australian Research Ethics Database (AU RED) for research ethics and governance processes, and thus refer to that database and its related measures. These will be updated to reflect the Research Ethics and Governance Information System (REGIS) in mid-2018.

NSW Health will periodically update the frequently asked questions for this metrics section for participating institutions. If you have any additional questions about the metrics or data collection, please get in touch with the Research Ethics and Governance team.

What are the data entry deadlines?

For 2017-18, please continue to enter data for metrics 1-6 using AU RED as outlined in Research Governance Metrics Data Collection and Reporting Manual. Further information about the collection of metrics in REGIS will be provided in advance of the REGIS launch.

What is the appropriate value to enter as “Maximum enrolment target per CTRA” for trials that don't have recruitment target at site?

Both Minimum and Maximum enrolment target per CTRA fields are designed as numeric data fields. Therefore, these fields only accept numbers and Maximum field must be greater than Minimum field. In the absence of a Maximum enrolment target, please record “9999”. This number will be used as a code to indicate that trial has no maximum enrolment target. NSW Health will clean these entries prior to reporting period.

What is a “valid reason” for not reaching an enrolment target, and for not enrolling a participant within 40 days of site authorisation? Furthermore, what is the definition of a rare disease, as indicated by one of the valid reason drop-down options?

Valid reason

One of the objectives of the metrics program is to help identify common barriers to commercial clinical trial recruitment. NSW Health provided a limited number of valid reason examples in AU RED to ensure that the data are ultimately site-generated. These qualitative data will in turn be used to inform future clinical research reform.

Rare disease

Per Rare Voices Australia, a rare disease is any disorder or condition that is a life-threatening or chronically debilitating disease which is statistically rare, with an estimated prevalence of 5 in 10,000 or of similarly low prevalence and high level of complexity that special combined efforts are needed to address the disorder or condition.

In light of that definition, institutions should rely on the clinical judgment of the reporting investigator to determine whether study enrolment was affected by a rare disease condition, even though its prevalence was accounted for during the study feasibility phase.

Ultimately, institutions are encouraged to work closely with trial sponsors to set achievable recruitment targets during the feasibility stage, which would include consideration of the patient population.

If there is a valid reason for not complying with a metric target, will those projects be removed from the overall calculation of the site’s performance?

NSW Health will assess the valid reasons provided and will communicate with institutions if there is a need for further clarification or discussion. If a reason is considered valid by NSW Health, the project will be removed from the metric’s denominator.

For metric 6, “sites reaching or exceeding their agreed enrolment target as per contract,” is a participant considered “enrolled” if she or he dies or withdraws/ doesn’t complete study?

A participant is considered enrolled as long as she or he is considered “enrolled” according to the protocol and under the terms of the CTRA.

The metrics manual implements new definitions for SSA application received date, initial clock start date and final clock stop date, for all applications authorised as of 1 July 2016. It also outlines new definitions for Post Approval Study State. Must institutions quality review and correct AU RED data for studies authorised before 1 July 2016?

Institutions may, but are not required to, review and update prior AU RED data based on the manual, with two important exceptions. To ensure clock data on all SSA applications that were authorised before 1 July 2016 remain comparable between institutions, institutions should not modify the clock start or stop dates for those studies based on the new clock date definitions.

How should those collecting the data communicate with researchers about the information needed?

Institutions are free to choose how best to communicate with their researchers. NSW Health will not be communicating data collection needs directly with researchers; however, institutions may share with researchers the related NSW Health website pages and materials.

The research governance grant should be allocated, at least in part, to putting systems in place to communicate about and gather these data from the relevant researchers/departments.

At a minimum, institutions will be collecting and recording the following information from researchers.

For all commercially-sponsored clinical trials greater than low risk authorised after 1 July

• Date first participant enrolled at the site.
• If there is a valid reason that the first participant was enrolled more than 40 calendar days after site authorisation, a description of the valid reason.

For all commercially-sponsored clinical trials greater than low risk closed to enrolment at the site after 1 July, regardless of when it was authorised

• A notification that the study has closed to enrolment at the site, ideally with the date that the study closed to enrolment at the site.
• Actual enrolment at the site by the end of the recruitment period/study closure. A participant is considered “enrolled” for data reporting purposes if the participant is considered “enrolled” per the Clinical Trial Research Agreement (CTRA).
• If there is a valid reason that the CTRA minimum recruitment target was not met, a description of the valid reason.

To help inform the metric on days to first participant enrolment, institutions may also request that researchers report the date of clinical trial site activation from investigators of – at least – all commercially-sponsored clinical trials greater than low risk, for all trials authorised within the reporting period.

Data collection system

How will the launch and implementation of the Research Ethics and Governance Information System REGIS affect the collection of metrics data?

NSW Health will formally manage the transition from AU RED to the REGIS system, and will communicate with the reporting institutions accordingly when additional information is available.

Data reporting

What are NSW Health’s plans for data review and reporting?

Following the data entry deadlines, NSW Health will communicate a database lock date, and will retrieve the data from AU RED at that time. NSW Health will evaluate the data quality to identify any inconsistent and missing data, and any apparent outliers. If required, a report and queries will be provided to each institution to correct data fields and/or complete missing data fields. After a data correction period, NSW Health will do a final database lock and will retrieve the data from AU RED for evaluation based on the metrics and the performance measures described above. Once the data analysis is completed, a final report will be provided to each institution.

NSW Health is currently working with Infonetica to determine the best way that institutions’ raw data can be provided to each institution for internal use and evaluation as well.