DoStrengthen your evaluation by using a mixture of data collection methods to enable you to triangulate your findings.
The following tools are either internal resources developed by the BNSSG ICB Clinical Effectiveness Research Team or external resources we have found useful.
BNSSG ICB Clinical Effectiveness and Research Team Guide to Types of data
BNSSG ICB Clinical Effectiveness and Research Team Guide to Questionnaires
BNSSG ICB Clinical Effectiveness and Research Team Guide to secondary data sources
BNSSG ICB Clinical Effectiveness and Research Team guide to interviews and focus groups
BNSSG ICB Clinical Effectiveness and Research Team Guide to case studies
Debriefing Engagement & Evaluation Participants
Best Practice in the Ethics and Governance of Service Evaluation
Commissioning Intelligence Model
BNSSG ICB Clinical Effectiveness and Research Team Data analysis guide
Midlands DSC Guide to Evaluation Design, Principles and Practice (sections on both qualitative and quantitative analysis)
Please note we are not responsible for the content of external sites and these are for guidance only.
What and Why
The next steps are to:
- Collect your data
- Analyse the data
How Will I Collect the Data?
A huge amount of data is collected routinely as part of delivering, monitoring and managing health and care services, for example, process data on activity like appointments, prescribing, admissions, referrals; data on structure such as staffing ratios, training levels; and data on outcomes like mortality, readmission, patient experiences. Any of this routinely collected data may be equally relevant to a service evaluation. If this is the case for your evaluation, think about who in your organisation may be able to help with this and:
- Identify what routine data is being collected by the organisation about its service and service users, including performance management data (which can be compared with national or regional data).
- Consider what data may be collected by other services which could be helpful (such as the police service, local councils, community services).
- Remember to consider this for helping you establish a baseline.
This may save you having to collect more data yourself.
If you decide you do need to collect new data to inform your evaluation, the two main types of data are quantitative (numbers) and qualitative (narrative) data. Different methods are used for collecting different types of data. Quantitative data collection methods give numerical results whereas qualitative data collection methods give narrative or descriptive data. Some common methods used in evaluation include:
- Quantitative methods: Questionnaires (administered electronically or sometimes, as a structured interview) and validated outcome measures such as measures of health and wellbeing. Using a validated tool saves you time and you can be sure that they are accurate and reliable measures.
- Qualitative methods: Interviews (mostly semi-structured), focus groups, observations and document reviews.
With either approach, sample size will be an important consideration. In simple evaluations that are local and relatively small scale, we often do not seek generalisable data and can settle for samples that do not aim to be representative. We can use common sense to judge how many is enough. But with larger evaluations, generalisability may be an important factor and in this case you should consult an expert on sampling (see box below).
There are further resources linked to qualitative and quantitative data collection methods in the resources section of the toolkit.
Collecting Economic Data
It may be appropriate to collect actual financial data, e.g., how much has been spent on a particular service over a specified period. This can entail including costs for a wide range of factors, e.g. staff time, consumables, equipment. At its most basic level, economic analysis could be for example, an hourly cost of e.g. a nurse, multiplied by the number of hours worked per day, month or year.
Economic data are always quantitative. There are measures widely used to assess value for money, e.g. the EQ-5D instruments give a result in QALYs (quality-adjusted life years), which allow a value to be assigned to quality of life. There are also proxy measures available which can be used to assign an economic value to a social factor, e.g., insomnia caused by worry. Useful reading about this can be found here and here. Economic analysis is likely to need expertise from an external source.
For further advice and guidance on collecting economic data and measuring costs and benefits visit the Evaluation Toolkit Economic Evaluation pages.
How Will I Analyse the Data?
Data analysis is a process by which the data you have collected is transformed into meaningful and useful information. It can be a complex process, with many possible approaches that are specific to the data collection methods used. This is where it can be particularly helpful to engage your experts and seek advice about how to analyse and interpret the data. You may also want to undertake some training in data analysis; links to local providers of training are in the toolkit including some online training.
A common approach to the analysis of qualitative data is thematic analysis which in very general terms involves looking for patterns or themes in the data. There are various ways to do this but a popular method is theframework method. This has clear steps to follow making it accessible to those who are new to qualitative analysis. There are further resources about analysing qualitative data in the Toolbox.
The analysis of quantitative data can range from simple counting of the data and describing what you have found (descriptive statistics) to more complex statistical analysis which determines whether you can infer from your data to a wider population (inferential statistics). Of course, this depends on the sample size you have been able to achieve and whether this is representative. Depending on your own knowledge and skills, the more complex analysis will possibly require the input and support of an experienced researcher, statistician or analyst. There is more information on data analysis in the Toolbox above and see also the list of experts.
For further advice and guidance on the analysis of economic data visit the Evaluation Toolkit Economic Evaluation pages.
- Public Health Data i.e. your Joint Strategic Needs Assessment (JSNA) can be accessed via Public Health Analysts in your Public Health teams within your local Council – find contact details here.
- Performance/contract or benchmarking data can be accessed from Performance analysts / contract managers.
- Statistician or Modelling support from an analyst or statistician you may find these in your ICB/NHS organisation, Public Health Team or University.
- Patient / User Experience and Satisfaction contact your Patient and Public Involvement (PPI) Lead within your organisation or People in Health West of England.
- Best Practice, Research and Evaluation Evidence – check out our EvidenceWorks toolkit for more information.
- Improvement and clinical audit data contact your Service/Quality Improvement Team or clinical audit department.
“When planning your evaluation, think about where you may be able to use the wealth of data that exists, particularly in health and social care, to feed into your evaluation. Match the existing data to your evaluation’s objectives before making a decision on whether to add new data”