Frequently Asked Questions (FAQs)

Evaluation forms part of the evidence base that helps us to work towards our population receiving ‘the right care, in the right place at the right time’ and is an intrinsic part of evidence informed commissioning and service provision.

Evaluation can help  to understand whether a service works, how it works, if not, why not, as well as identify intended or unintended outcomes and areas for improvement.

There is considerable evidence that the findings of evaluations often influence policy, and service planning and implementation (Rossi, Lipsey and Freeman 2019). This means evaluation is a very valuable tool. The knowledge and information gained can benefit your service, patients and service users, staff, health and social care commissioners and the NHS as a whole. The key benefits are that the evaluation informs service planning, provides evidence of success and gives feedback to all those involved.

An important question to ask yourself is: “Can I afford not to do an evaluation?” (Magenta Book, 2011).

A common question is how much of the budget should be spent on the evaluation. This very much depends on the purpose and nature of the evaluation. However, we tend to recommend that for projects in the thousands allocate approximately 5-10% of the total value towards the evaluation, those in the millions tend to be more towards the region of 3-5%.

For example a small pilot service worth around £20,000 would allocate £1-2000 whereas a £10million pound service would allocate nearer £400,000. This will also impact on the type of evaluation that is conducted, with the small scale pilot project likely to conduct an internal evaluation utilising existing resources whereas the large scale service is more likely to conduct an independent evaluation using external resources.

This reflects similar recommendations from the World Health Organisation and the National Lottery Community Fund.

There are a number of people in your own organisation who may be able to help you, these include your Research, Development and/or evaluation team (don’t forget if you work for Bristol, North Somerset or South Gloucestershire ICB to come and see the BNSSG ICB Clinical Effectiveness and Research Team, Quality Improvement leads or programme and project managers experienced in evaluation.

You may also want to seek external help from experts and there are a number of local networks and organisations that may be able to help you including NIHR Applied Research Collaboration (ARC) West,  West of England Academic Health Science Network (AHSN), Bristol Health Partners Academic Health Science Centre, Evaluation Online Network, or your local University.

You may however which to procure an external independent evaluation, if so see FAQ 4.

You will need to comply with your own organisations procurement processes and so we recommend that you contact your procurement lead/team in the first instance as they will be able to advise. We recommend that you also engage an evaluation expert as a part of the procurement process.

We recommend you are not prescriptive regarding the methodology so that those bidding to provide the evaluation can demonstrate their expertise and allows for appropriate innovation. However, use your organisation’s procurement team as they will ensure that you comply with all the legal and organisational requirements.

In terms of contracts, if it is for an NHS service you will generally use the NHS Terms and Conditions for the Provision of Services.

The short answer is yes. You will need to seek agreement from the organisations and people involved. This may be through formal governance systems or by seeking informed consent from those who are being asked to participate.

See our Review and Act page as they provide useful tools to help you. Speak to your communications team and see how they can help you as well as use resources from other local initiatives, such as Knowledge Mobilisation (a partnership between the BNSSG ICB Clinical Effectiveness and Research Team, University of Bristol and University of the West of England, Bristol).  For more information please the ‘What is Evaluation?’ page.

There are several recognised approaches that can be adopted to help identify benefits. These include the use of logic models, stakeholder “benefits workshops”, referring to case studies of similar initiatives adopted elsewhere, or discussing benefits with patients

Good benefits management should begin as early as possible in the programme or project, which will often be significantly before implementation has commenced to enable the baseline to be captured.

The most common type of baselining is pre and post implementation which simply compares the change in the benefit metric before and after the business change has been introduced. When deciding on a baseline a range of influencing factors should be considered including the date of implementation; any phasing of the rollout; the feasibility of analysing a suitable comparator; the availability of data prior to delivery; and any confounding variables which may impact the ability to draw accurate conclusions from the data.