This is an important document because it describes the challenges faced with existing evaluation methodology, and proposes a new approach to measuring the impact of new, complex models of care. It is aimed at anyone who is involved in service redesign in the NHS, and is a comprehensive guide to this innovative methodology. Throughout the guide, the authors demonstrate how retrospective matched control analysis can be used to evaluate the clinical and cost effectiveness of one way of working compared with another.

Overview of potential evaluation methods

The Nuffield Trust defines Retrospective matched control analysis (RMCA) as “routinely collected data is used to construct a matched control group, and the impact of the intervention (or service redesign) is measured in terms of differences in the outcomes relative to the matched control group.” In other words, RMCA is when outcomes such as unexpected hospital admissions, are compared with groups of patients who were treated differently. Patients are matched for demographic data, such as age, sex, socio-economic factors, existing conditions, etc. It helps people who design services to see which care model was most effective, but has a high risk of bias, which is important for effective planning of services.

The authors of this guide have compared this method with other evaluation methods, such as:

• Randomised controlled trials, where there are issues with recruiting patients.
• Longitudinal studies, comparing populations over time, where some outcomes may occur, regardless of the care model applied.
• Ecological studies, looking at a local geographical area, which might have different outcomes to a wider population.

They list the benefits and limitations of using RMCA, and conclude that it is a more robust method to use for evaluation purposes. RMCA uses existing data, routinely collected within hospitals, which means that patients are not actively recruited, and the outcomes have already occurred. Therefore the evaluation can take place behind the scenes, with no need of direct patient involvement.

Ten steps towards retrospective matched control analysis

To facilitate the application of this method of evaluation, the guide outlines ten key steps that evaluators need to follow:

1. Clarify the aims of the service and the evaluation – it is important that there is a purpose to the evaluation and that everyone understands what that purpose is.
2. Decide on the number of people needed to demonstrate an effect – this point goes into great detail about power calculations, which provide an estimate of how many people must be included to get a significant result.
3. Ensure permission is granted to access person-level datasets – information governance is extremely important in the NHS, to ensure the privacy of service users. Either permission must be granted or anonymised data should be used.
4. Ensure there are data on who received the new service, and some information about the service received – if this data is not available, the evaluation process will not glean useful results.
5. Identify the potential control population – in this step, the authors provide useful guidance describing the choices for controls available.
6. Create longitudinal patient-level histories of service use – this step describes how best to do this, to ensure that a useful amount of data is collected to present a clear picture of the history and care activity over time.
7. Identify matched controls – while the authors go into great detail about how to do this, they do recommend that this task should be undertaken by a specialist analyst, someone with expertise in carrying out this sort of research methodology.
8. Monitor outcome variables for those receiving the new service and matched controls – this means referring back to step 1, and seeing what the initial outcomes are. The guide provides an example of how the Nuffield Trust has carried out this step.
9. Undertake summative analysis – again, the authors have provided an example on how to do this, but specialist expertise would make the process easier, and make the results more robust.
10. Continuously monitor – here the authors explain why it is so important to continuously monitor the situation. Health systems are constantly change, and therefore, continuous evaluation is necessary, to ensure that current methods are still the most appropriate.

Commentary

Evaluation methodology is vital to service redesign, as it provides the rationale for making changes to existing care models. Published international research evidence of new models can inform policymakers and commissioners about potential new care models, but this evaluation methodology, carried out in the relevant populations, will ensure that the service is designed to suit local, regional, and national needs. The authors of this guide have provided the theory, alongside practical steps and examples to help you implement this evaluation methodology. They have provided the argument for using RMCA above other study methods, and provided a clear description of the process.

When implementing this methodology, step 7 should perhaps be thought of nearer the start of the process, because without specialist expertise in this area, it will be difficult to carry out this research in a robust manner. Within your areas of responsibility, do you have access to experts in the areas of research methodology and data analysis? Perhaps this is something to think about when undertaking an evaluation project using RMCA methodology.

Particularly with so many stakeholders involved in the commissioning of new services, it is essential that rigorous, unbiased data is available for all decision-makers to see, so that the right decision can be made, with regards to service redesign.

Link

Davies A, Ariti C, Georghiou T, Bardsley M. (2015) Evaluation of complex health and care interventions using retrospective matched control methods. Nuffield Trust: London

Related links

Nuffield Trust – Evaluation