Comment

Implementing and evaluating behaviour change programmes with the Public Sector Scorecard

Max Moullin and Rob Copeland explain how the Public Sector Scorecard can be used to evaluate public health programmes.

Achieving the objectives of many public and voluntary sector organisations involves changing people’s behaviour. Whether they wish people to drink less, drive more carefully, reduce substance abuse, lose weight, improve sexual health, commit less crime, apply for jobs, or stop smoking, they cannot actually control whether people make such changes. So how can organisations make sure that their strategies will actually change people’s behaviour in the direction they want? Also how will they know whether they have been successful and importantly how this success was achieved?

These were the questions asked when the two authors came together to evaluate Sheffield Let’sChange4Life, a £10m programme addressing obesity in children and families in the city, part-funded by the Department of Health. The resultant collaboration led to a significant development in addressing and evaluating behaviour change programmes by incorporating the Theory of Planned Behaviour (Ajzen, 1991) into the Public Sector Scorecard (Moullin, 2002).

The Public Sector Scorecard

The Public Sector Scorecard (PSS) is an integrated service improvement and performance management framework that adapts and extends the balanced scorecard for the public and third sectors. The PSS has been used in central and local government as well as healthcare and is consistent with Lord Darzi’s recommendation that “NHS services must develop quality frameworks that combine relevant indicators, defined nationally, with those appropriate to local circumstances.”

Originally developed in 2001, the Public Sector Scorecard (PSS) is a workshop-based approach working with managers, staff, service users and other key stakeholders. The model focuses on the outcomes that matter to service users, the processes that deliver these outcomes, and the organisation’s capability to support its people and processes to achieve the relevant outcomes.

The PSS has three phases – strategy mapping, service improvement and measurement and evaluation.

Strategy mapping

This involves identifying the outcomes that matter to service users and other key stakeholders, including value for money, in a workshop setting, together with the outputs that the various processes involved need to achieve in order to deliver these outcomes. Following this the group would be asked to identify the capability outputs that are needed to ensure that staff and processes are able to achieve the outcomes and process outputs required. These aspects might include effective team and partnership working, sufficient resources, and supportive leadership.

The links between capabilities, processes and outcomes are then illustrated in a strategy map, which is an important intermediate output of the PSS. The draft strategy map will then be refined following a risk management workshop by considering the reduction of a key risk as a desired outcome.

The processes by which risks are reduced, eliminated or mitigated are then reviewed, together with the risk management culture, and added to the strategy map.

Service improvement

In this phase the strategy map is used as a prompt to examine how effective the current processes are in achieving the outcomes and outputs and how they can be improved. The views of workshop participants will be supplemented where appropriate by using tools such as process maps, systems thinking and lean management – for example to highlight areas of duplication, processes that could be simplified or accelerated through better communication, and eliminating nonproductive activities such as talking to patients who ring up because they have not received a service they were promised (Moullin, 2008).

In addition, with the help of workshop participants, the organisation needs to focus on addressing the capability outputs in the strategy map and in particular how management can support staff and processes so that they can achieve the outcomes required.

This could involve extra resources in a particular area, improving staff morale, and clear supportive leadership. It might also involve ensuring a culture of improvement and innovation rather than a blame culture, which is essential to make full use of the PSS.

Measurement and evaluation

This phase begins by identifying possible performance measures for each element of the strategy map. These will be filtered to take account of data quality issues and to ensure that potential perverse effects are minimised. Analysing and learning from performance measures provides insight into how well organisations are performing in the different areas of the strategy map. Taking action to address areas needing attention is also needed!

Using the Public Sector Scorecard to evaluate behaviour change programmes

As with the balanced scorecard, the strategy map is a key output of the PSS. However while Kaplan and Norton (2001) define the strategy map as “describing how shareholder value is created from intangible assets”, with the Public Sector Scorecard it is described more simply as “depicting the relationships between capability, process and outcome elements” (Moullin, 2009).

It follows therefore that a well-designed strategy map for a behaviour change programme such as reducing obesity needs to incorporate the factors that will influence behaviour change.

This was recognised in the evaluation of Sheffield Let’s Change4Life. Only if we could establish the success or otherwise of the various activities in the programme in addressing the factors that influence change would we be able to provide the insights required. The approach used was to incorporate the evidence-based Theory of Planned Behaviour into the Public Sector Scorecard.

The Theory of Planned Behaviour

The Theory of Planned Behaviour has often been used to study health-related decision making in adults and young people. It has also been found effective as a model for predicting physical activity behaviour. The theory suggests that the immediate determinant of behaviour is intention, but this is directly driven by three major constructs: attitude, subjective norm, and perceived behavioural control.

Attitude is defined as the degree to which an individual has a favourable or unfavourable perception of the behaviour. Subjective norm refers to the perceived importance others hold about performing or not performing a given behaviour and one’s willingness to comply to those referents. Perceived behavioural control describes the perceived ease (confidence to perform) or difficulty (perceived barriers to overcome) an individual has for performing a given behaviour.

Incorporating the Theory of Planned Behaviour into PSS Strategy Maps

The main relevance of the TPB for strategy mapping is that when developing a strategy for a programme or organisation that aims to support people to change their behaviour, it is important for the programme to address a number of issues simultaneously: people’s beliefs on how important it is to make the change, their attitude and those of others around them, their perceived ability to make the change, and overcoming the barriers that they face. Strategy maps were developed both for the Sheffield Let’s Change4Life programme as a whole and for each of its eight strands.

Figure 3, below, shows how the TPB was incorporated into the strategy map for the strand promoting breastfeeding. This strand included two activities – a peer support scheme and breastfeeding-friendly awards – shown in the bottom row. The row above corresponds to attitude, subjective norm, and perceived behavioural control from the TPB. Progress on these elements was identified as being likely to result in more women intending to breastfeed.

This strategy map helped the programme focus on each of the various areas which will influence behaviour. It also aided the evaluation by supplementing the data on breastfeeding maintenance rates by asking the women whether they were more positive about the idea of breastfeeding, whether they saw it as a socially approved behaviour, and whether they felt more confident about being able to breastfeed when the time came.

Another example of the usefulness of the approach was when evaluating a workshop on diet and exercise provided for workers at a Sheffield steel manufacturer.

While feedback from participants was quite positive, one senior manager commented that the company did not have space for a canteen on site – but there was a convenient mobile burger bar parked outside the factory gates every lunchtime!

Clearly, unless that particular barrier is overcome, a single information giving workshop would be unlikely to lead to a significant change in obesity levels of workers or their families.

Overall Strategy Map for Sheffield Let’s Change4Life

The strategy map for the Sheffield Let’s Change4Life (SLC4L) programme as a whole is shown below. This was developed following interactive workshops with the Programme Board, operational leads and stakeholders of the eight programme strands, and Sheffield Youth Council.

Rows A and B show the main outcomes required for the project. The main desired outcome was to reduce obesity, while other key outcomes which will contribute towards this overall outcome include better diet and nutrition and increased physical activity.

Satisfied stakeholders, sustainability and value for money were also key aims.

Row C contains the Theory of Planned Behaviour outputs and outcomes: a greater desire to adopt a healthy lifestyle; favourable attitudes; confidence in their ability to change; and overcoming the barriers they face – while elements D1 to D8 refer to the desired outcomes and outputs of the eight strands of the programme.

The penultimate row, row E, shows the main elements that need to be in place to support individual strands including joint working and a shared vision between Sheffield City Council, the NHS, and the third and private sectors, community engagement, and effective project management, all of which need to be underpinned by effective leadership and support from the programme board (row F).

Using the Public Sector Scorecard to evaluate SLC4L

The strategy map was used to help managers and strand leads focus on the desired outcomes, to understand and explore with them how change might be expected to happen, and to monitor and evaluate performance. The evaluation team assessed the impact of Sheffield Let’s Change4Life on each of the elements of the strategy map, using a variety of measures and approaches. The theory of planned behaviour outcomes C1-C4 were assessed using questionnaires asking participants before and after a intervention on each of these aspects in turn.

The approach enabled the evaluation team to supplement the nationally available data on obesity rates with information on the impact of the programme on factors which are thought likely to make an impact on future obesity rates.

This is particularly useful as the time lag between taking part in and realising the health benefits from ‘prevention-based’ activities often means that short evaluation follow-up periods miss the real impact of the programme.

The evaluation commended the systems approach adopted by SLC4L, its strong leadership and joint working between public health and the city council, and the impact on participants’ attitudes and intentions towards the adoption of health promoting behaviours. In addition, latest data from the National Child Weight Measurement Programme in Sheffield suggest a marginal fall in the prevalence of overweight and obesity in children at year 6 (age 10-11).

A particular benefit of the PSS is that it facilitated joint working between the NHS, local government and the third and private sectors by encouraging staff to focus on common desired outcomes rather than their own organisational objectives.

Sheffield City Council’s executive director for Children, Young People and Families said that ‘the strategy map is really useful as it simplifies a complex issue with a complex response into an orderly understandable approach’, while the programme director of SLC4L commented:

“The SLC4L Strategy Map was a very useful tool in terms of explaining and evaluating the programme. The format of the map was easy to understand and was used to great success with deliverers of the programme, as well as other stakeholders, leadership across the City, and the public. It visually told the story of SLC4L, what we were trying to achieve and how.

“It also helped all those involved understand the outcome and process measures the programme was trying to achieve, and therefore being evaluated against. It provided an ‘at a glance’ understanding of SLC4L.”

Conclusion

Health and social care organisations need not only to clarify the outcomes they are looking for and ensure that their processes are well designed, but they also need to address the resource, partnership working, and leadership issues required to support their staff. Also important is developing a culture of improvement, innovation and change rather than a blame culture. With its emphasis on outcomes, processes and capability, the Public Sector Scorecard is an ideal framework to help organisations achieve their outcomes, including value for money, and evaluate how well they have performed.

Organisations seeking to achieve behaviour change — for example those aiming to reduce obesity, alcohol, smoking or drug use — need also to address the various factors that are likely to influence change.

For these organisations, the approach described in this paper, integrating the Theory of Planned Behaviour into the Public Sector Scorecard, will help them develop their strategies, improve services, and evaluate performance.

This approach provides the missing link between outputs and outcomes for behaviour change as it provides a pathway to understand how change might or indeed does occur.

About the authors

Max Moullin is director of the Public Sector Scorecard Research Centre, a visiting fellow at Sheffield Business School, and a member of the Healthcare Advisory Forum.

Dr Robert Copeland is a chartered psychologist and Reader in physical activity and health at Sheffield Hallam University’s Centre for Sport and Exercise Science.

(Top image: Sheffield breastfeeding campaign)