Do

 Steps

The next steps are to:

1. Collect your data
2. Analyse the data

How Will I Collect the Data?

Routine Data

A huge amount of data is collected routinely as part of delivering, monitoring and managing health and care services, for example, process data on activity like appointments, prescribing, admissions, referrals; data on structure such as staffing ratios, training levels; and data on outcomes like mortality, readmission, patient experiences. Any of this routinely collected data may be equally relevant to a service evaluation. If this is the case for your evaluation, think about who in your organisation may be able to help with this and:

• Identify what routine data is being collected by the organisation about its service and service users, including performance management data (which can be compared with national or regional data).
• Consider what data may be collected by other services which could be helpful (such as the police service, local councils, community services).
• Remember to consider this for helping you establish a baseline.

This may save you having to collect more data yourself.

Primary Data

If you decide you do need to collect new data to inform your evaluation, the two main types of data are quantitative (numbers) and qualitative (narrative) data. Different methods are used for collecting different types of data. Quantitative data collection methods give numerical results whereas qualitative data collection methods give narrative or descriptive data. Some common methods used in evaluation include:

• Quantitative methods: Questionnaires (administered electronically or sometimes, as a structured interview) and validated outcome measures such as measures of health and wellbeing. Using a validated tool saves you time and you can be sure that they are accurate and reliable measures.
• Qualitative methods: Interviews (mostly semi-structured), focus groups, observations and document reviews.

With either approach, sample size will be an important consideration. In simple evaluations that are local and relatively small scale, we often do not seek generalisable data and can settle for samples that do not aim to be representative. We can use common sense to judge how many is enough. But with larger evaluations, generalisability may be an important factor and in this case you should consult an expert on sampling (see box below).

There are further resources linked to qualitative and quantitative data collection methods in the resources section of the toolkit.  

Collecting Economic Data

It may be appropriate to collect actual financial data, e.g., how much has been spent on a particular service over a specified period. This can entail including costs for a wide range of factors, e.g. staff time, consumables, equipment. At its most basic level, economic analysis could be for example, an hourly cost of e.g. a nurse, multiplied by the number of hours worked per day, month or year.

Economic data are always quantitative. There are measures widely used to assess value for money, e.g. the EQ-5D instruments give a result in QALYs (quality-adjusted life years), which allow a value to be assigned to quality of life. There are also proxy measures available which can be used to assign an economic value to a social factor, e.g., insomnia caused by worry. Useful reading about this can be found here and here. Economic analysis is likely to need expertise from an external source.

For further advice and guidance on collecting economic data and measuring costs and benefits visit the Evaluation Toolkit Economic Evaluation pages.

How Will I Analyse the Data?

Data analysis is a process by which the data you have collected is transformed into meaningful and useful information. It can be a complex process, with many possible approaches that are specific to the data collection methods used.  This is where it can be particularly helpful to engage your experts and seek advice about how to analyse and interpret the data. You may also want to undertake some training in data analysis; links to local providers of training are in the toolkit including some online training.

A common approach to the analysis of qualitative data is thematic analysis which in very general terms involves looking for patterns or themes in the data. There are various ways to do this but a popular method is the framework method. This has clear steps to follow making it accessible to those who are new to qualitative analysis. There are further resources about analysing qualitative data in the Toolbox.

The analysis of quantitative data can range from simple counting of the data and describing what you have found (descriptive statistics) to more complex statistical analysis which determines whether you can infer from your data to a wider population (inferential statistics). Of course, this depends on the sample size you have been able to achieve and whether this is representative. Depending on your own knowledge and skills, the more complex analysis will possibly require the input and support of an experienced researcher, statistician or analyst. There is more information on data analysis in the Toolbox above and see also the list of experts.

For further advice and guidance on the analysis of economic data visit the Evaluation Toolkit Economic Evaluation pages.