4Do

Steps

The next steps are to:

1. Collect your data
2. Analyse the data

How will I collect the data

Routine data

A huge amount of data is collected routinely as part of delivering, monitoring and managing health and care services. This includes data on appointments, prescribing, admissions, referral, staffing ratios, training levels, mortality, readmission or patient experiences.

Any of this routinely collected data may be equally relevant to a service evaluation. If this is the case for your evaluation, think about who in your organisation may be able to help with this and:

• Identify what routine data is being collected by the organisation about its service and service users, including performance management data (which can be compared with national or regional data)
• Consider what data may be collected by other services which could be helpful (such as the police service, local councils, community services)
• Remember to consider this for helping you establish a baseline

This may save you having to collect more data yourself.

Primary data

If you decide you do need to collect new data to inform your evaluation, the two main types of data are quantitative (numbers) and qualitative (narrative) data.

Different methods are used for collecting different types of data. Quantitative data collection methods give numerical results whereas qualitative data collection methods give narrative or descriptive data.

Some common methods used in evaluation include:

• Quantitative methods: Questionnaires (administered electronically or sometimes, as a structured interview) and validated outcome measures such as measures of health and wellbeing. Using a validated tool saves you time and you can be sure that they are accurate and reliable measures
• Qualitative methods: Interviews (mostly semi-structured), focus groups, observations and document reviews

With either approach, sample size will be an important consideration. In simple evaluations that are local and relatively small scale, we often don’t seek generalisable data and can settle for samples that do not aim to be representative.

We can use common sense to judge how many is enough. But with larger evaluations, generalisability may be an important factor and in this case you should consult an expert on sampling (see box below).

There are further resources linked to qualitative and quantitative data collection methods in the resources section of the toolkit.

Collecting economic data

It may be appropriate to collect actual financial data, for example how much has been spent on a particular service over a specified period. This can mean including costs for a wide range of factors, such as staff time, consumables or equipment. At its most basic, economic analysis could be an hourly cost of a nurse, multiplied by the number of hours worked per day, month or year.

Economic data is always quantitative. There are measures widely used to assess value for money, such as the EQ-5D instruments which give a result in QALYs (quality-adjusted life years), allowing a value to be assigned to quality of life.

There are also proxy measures available which can be used to assign an economic value to a social factor, for example insomnia caused by worry. Useful reading about this can be found on the Social Value and UK Social Value Bank websites. Economic analysis is likely to need expertise from an external source.

For further advice and guidance on collecting economic data and measuring costs and benefits visit the economic evaluation pages.

How will I analyse the data?

Data analysis is a process by which the data you have collected is transformed into meaningful and useful information.

It can be complex, with many possible approaches that are specific to the data collection methods used. This is where it can be particularly helpful to engage your experts and seek advice about how to analyse and interpret the data.

You may also want to undertake some training in data analysis. Links to local providers of training are in the toolkit including some online training.

A common approach to the analysis of qualitative data is thematic analysis which involves looking for patterns or themes in the data. There are various ways to do this but a popular method is the framework method. This has clear steps to follow making it accessible to those who are new to qualitative analysis. There are further resources about analysing qualitative data in the toolbox.

The analysis of quantitative data can range from simple counting of the data and describing what you have found (descriptive statistics) to more complex statistical analysis which determines whether you can infer from your data to a wider population (inferential statistics). Of course, this depends on the sample size you have been able to achieve and whether this is representative.

Depending on your own knowledge and skills, the more complex analysis will possibly require the input and support of an experienced researcher, statistician or analyst. There is more information on data analysis in the toolbox above and see also the list of experts.

For further advice and guidance on the analysis of economic data visit the economic evaluation pages.

Local experts

• Public health data: your Joint Strategic Needs Assessment (JSNA) can be accessed via public health in your local council
• Performance, contract or benchmarking data can be accessed from performance analysts and contract managers
• Statistician or modelling support from your ICB or other NHS organisation, public health team or university
• Patient and user experience and satisfaction contact your patient and public involvement (PPI) lead in your organisation
• Best practice, research and evaluation evidence – check out our evidence toolkit for more information
• Improvement and clinical audit data: contact your service or Quality Improvement team or clinical audit department