FAQ

Evaluation forms part of the evidence base that helps us to work towards our population receiving ‘the right care, in the right place at the right time’ and is an intrinsic part of evidence informed commissioning and service provision.

Evaluation can help to understand whether a service works, how it works, and if not, why not, as well as identify intended or unintended outcomes and areas for improvement.

There is considerable evidence that the findings of evaluations often influence policy, and service planning and implementation (Rossi, Lipsey and Freeman 2019). This means evaluation is a very valuable tool. The knowledge and information gained can benefit your service, patients and service users, staff, health and social care commissioners and the NHS as a whole. The key benefits are that the evaluation informs service planning, provides evidence of success and gives feedback to all those involved.

An important question to ask yourself is: “Can I afford not to do an evaluation?” (Magenta Book, 2011).

A common question is how much of the budget should be spent on the evaluation. This very much depends on the purpose and nature of the evaluation. However, we tend to recommend that for projects in the thousands allocate approximately 5-10% of the total value towards the evaluation, those in the millions tend to be more towards the region of 3-5%.

For example a small pilot service worth around £20,000 would allocate £1-2000 whereas a £10million pound service would allocate nearer £400,000. This will also impact on the type of evaluation that is conducted, with the small scale pilot project likely to conduct an internal evaluation utilising existing resources whereas the large scale service is more likely to conduct an independent evaluation using external resources.

This reflects similar recommendations from the World Health Organisation and the National Lottery Community Fund.

There are a number of people in your own organisation who may be able to help you, these include your research, development or evaluation team (don’t forget if you work for Bristol, North Somerset and South Gloucestershire ICB you can talk to the BNSSG ICB Clinical Effectiveness team), Quality Improvement leads or programme and project managers experienced in evaluation.

You may also want to seek external help from experts and there are a number of local networks and organisations that may be able to help you including the NIHR Applied Research Collaboration West (NIHR ARC West), Health Innovation West of England, Bristol Health Partners Academic Health Science Centre, Evaluation Online Network, or your local university.

Or you may wish to procure an external independent evaluation. If so see the next FAQ.

You will need to comply with your own organisation's procurement processes and so we recommend that you contact your procurement lead or team in the first instance as they will be able to advise. We recommend that you also engage an evaluation expert as a part of the procurement process.

We recommend you are not prescriptive regarding the methodology so that those bidding to provide the evaluation can demonstrate their expertise and allow for appropriate innovation. However, use your organisation’s procurement team as they will ensure that you comply with all the legal and organisational requirements.

In terms of contracts, if it is for an NHS service you will generally use the NHS terms and conditions for the provision of services.

The short answer is yes. You will need to seek agreement from the organisations and people involved. This may be through formal governance systems or by seeking informed consent from those who are being asked to participate.

Review and act has useful tools for sharing your evaluation findings.

Speak to your communications team and see how they can help you. If you have knowledge mobilisation experts in your organisation, talk to them too.

In Bristol, a knowledge mobilisation (KM) initiative is evolving to support access to new research findings and promote ‘research informed commissioning’ and ‘commissioning informed research’.

This started as a small team of academics and commissioning managers (the KM Team) co-ordinated by BNSSG Clinical Effectiveness team and the University of Bristol. It's now expanding to include University of the West of England and embraces other KM-related roles.

For more information see about evaluation.

There are several recognised approaches that you can use to help identify benefits. These include logic models, stakeholder “benefits workshops”, case studies of similar initiatives adopted elsewhere or discussions with patients.

Good benefits management should begin as early as possible in the programme or project. This will often be significantly before implementation has started to enable the baseline to be captured.

The most common type of baselining is pre and post implementation. This simply compares the change in the benefit metric before and after the business change has been introduced.

When deciding on a baseline, you should consider a range of influencing factors including the date of implementation, any phasing of the rollout, the feasibility of analysing a suitable comparator, the availability of data prior to delivery and any confounding variables which could impact the ability to draw accurate conclusions.