Tag Archive for: Evaluation Tools

What Happens After You Appoint an External Evaluator?

What happens after you appoint The Evaluator?

Appointing an external evaluator can feel like handing an important part of your project to people you do not yet know very well. You may be wondering how much time the process will take, what your team will be expected to do, whether the evaluator will understand the work quickly enough, and whether problems will be spotted before the final report.

Those are reasonable concerns. A strong proposal matters, but clients also need confidence in what happens after the contract is signed.

Our job is to make evaluation feel held: clear responsibilities, sensible tools, regular contact and enough challenge behind the scenes to produce findings that stand up to scrutiny. This is the process you can expect from us.

1. We begin by getting the important things out of people’s heads

The first stage is inception. We review the brief, funding requirements, project plans, existing monitoring and any previous research. Then we talk through what the project is really trying to achieve, what is already happening and where the uncertainties sit.

This is often where different strands begin to join up. Projects can involve several partners, audiences, activities, outcomes and reporting requirements. We create a shared picture, agree priorities and make decisions about what the evaluation can realistically answer.

This is often the point when people start to feel excited about the evaluation: the project becomes clearer, the possibilities begin to emerge, and the team can see how the process will support their work. We also begin agreeing the practical arrangements and responsibilities, building these out as the evaluation develops.

2. We turn ambition into answerable questions

Funding applications often contain broad and overlapping ambitions. Our role is to translate those into a focused set of evaluation questions and, where useful, a theory of change.

That means identifying the route from activity to outcomes, the assumptions underneath it and the evidence that will help us judge progress. We also review your targets and deliverables, checking that everyone understands what needs to be counted, how it will be recorded and who will be responsible. Where possible, we establish baselines or sensible comparison points and make clear where evidence will be indicative rather than conclusive.

This work creates the spine of the evaluation. Without it, projects can collect a great deal of information and still struggle to demonstrate what they delivered or answer the questions that matter.

3. We build tools that work in the real world

Next, we develop the evaluation framework and practical toolkit. This could include monitoring templates, surveys, interview guides, observation tools, participant reflection, demographic questions or creative in-person methods.

The right tool depends on the setting. A short community activity, an outdoor festival, a long-term volunteer programme and a strategic partnership cannot all be evaluated in the same way.

We design first for the people who will actually use and respond to the tools, as well as for the evidence the evaluation needs. If you are working with people experiencing deprivation, people facing language barriers, children or people with complex disabilities, we will adapt both the method and the way it is delivered so that participation is practical, appropriate and meaningful.

That might mean using simpler language, visual or creative approaches, supported conversations, shorter questions or accessible alternatives to a conventional survey. We aim to create methods that work in real settings—not polished, bureaucratic tools that look impressive on paper but place unnecessary demands on participants or delivery teams.

4. We make responsibilities manageable

External evaluation works best as a clear partnership. Your team may hold attendance records, capture routine feedback or introduce us to participants. We may lead deeper interviews, focus groups, observation, analysis and independent reporting.

We agree that division early and provide support where needed. This might include a short training session, written guidance, worked examples or a regular check-in. The aim is not to turn delivery staff into researchers. It is to make the information they gather consistent and useful without adding unnecessary weight to their jobs.

5. We check the evidence while the project is live

One of the biggest risks in evaluation is waiting until the end to discover that an important question was not asked or that data has been recorded inconsistently.

We build in proportionate progress and data-quality checks. We look at what is coming in, whether different groups are being heard, where response rates are weak and whether delivery changes affect the original framework. If something is not working, we raise it while there is still time to adapt.

These conversations are not about catching clients out. They are about protecting the final evaluation and helping the project learn during delivery.

6. We analyse, compare and challenge

Analysis is more than summarising a spreadsheet or selecting good quotations. We look for patterns, differences between groups, changes over time and explanations for what we are seeing. We compare monitoring data, surveys, interviews, observation and other sources rather than treating each in isolation.

We also look for evidence that complicates the emerging story. Where samples are limited, baselines are missing or other factors may have contributed to change, we say so. Conclusions are tested within the wider team before they become part of the report.

That internal challenge is a core part of our quality assurance. Accessible writing is the final layer; underneath it sits careful analysis and professional judgement.

7. We turn findings into something people can use

The report should give funders and boards confidence, but it should also help the people delivering the work. We structure findings around the questions and outcomes that matter, show how conclusions were reached, and make limitations clear without allowing them to obscure useful learning.

Recommendations are grounded in the evidence and proportionate to the organisation’s capacity. Depending on the commission, we may also present findings, run a learning session or produce concise visual material for wider audiences.

Our aim is not to leave a large document behind and disappear. It is to help the right people understand what the evidence means and what to do next.

What happens when projects change?

Projects change because funding is delayed, staff move, activities evolve, partners join or audiences respond differently from expected. Evaluation has to be structured without becoming rigid.

We keep the central questions in view, record agreed changes and adapt tools or timing where that protects the usefulness of the work. If a change affects scope or budget, we explain that early rather than allowing a surprise to appear at the end.

What about capacity and continuity?

Clients appoint The Evaluator as a consultancy, not an isolated individual. Each project has clear leadership and project management, with wider team involvement in delivery, analysis, review and cover. Our systems hold actions, deadlines and decisions so that continuity does not rely on one person remembering everything.

That team structure matters. It gives clients consistent contact while bringing more than one perspective to important findings. It also means quality and progress can be protected through busy periods, leave and the ordinary realities of working life.

What we need from you

Good evaluation does require collaboration. We will need timely access to relevant documents and data, honest conversations about delivery, help reaching participants or stakeholders, and enough notice of key events and changes.

In return, we take responsibility for holding the evaluation: keeping the framework coherent, making sense of evidence, identifying risks, asking the difficult questions and turning complexity into clear, useful findings.

If you are planning an evaluation and want to understand what the process would look like for your project, send us your brief—or talk to us before the brief is finished. We will help you work out a proportionate route from the questions you have now to evidence you can confidently use.

Simple evaluation you can do yourself – Three tips to do your own evaluation

Sometimes we get asked to quote for projects and the project is quite small, or the project team is quite small. They don’t have a large budget and although we’re always great value for money, sometimes it’s better to spend the money on delivery.

Our Director, Kirsty, used to run small projects like this and always carried out her own evaluation. She used this knowledge to create ‘The Evaluator’ and wanted to put together a guide for someone who has got a bit of public funding and a small project (maybe £50,000 or less).

If your project is small we recommend you do your own evaluation. There are always exceptions to the rule however and sometimes the project has a strong learning objective, which makes an evaluation particularly appropriate, but as a guide – that’s what we’d recommend.

Say you are running a project and you’ve got, for example  £35,000 worth of funding -how would you attempt to evaluate that? There are some really very simple things that you could do, as follows:

Idea 1: Have an internal focus group.

For example, you might put a meeting together and the three or four people who have been running the project and working on most of the tasks, might sit down and answer these three questions:

What went really well?

What surprised you about the people taking part?

What would you do differently?

Just that alone – having a discussion and applying it, is a really good start.

Idea 2: Mini data analysis.

You keep a note of attendance at events and look to see which days are busiest and do more of them.

Idea 3: Mini research interviews.

Ask the people taking part what they have enjoyed the most – then plan to repeat that for new people.

The important thing is to not see ‘evaluation’ as a big scary thing, but to see it as ‘how can we use the knowledge or information we have to improve’.

Good luck!

VISIT IN PERSON: WE INVENTED THE WEEKEND

We were lucky enough to be invited to evaluate the ‘We Invented the Weekend’ festival at Salford Quays in June 2024. The festival is busy! It attracts thousands of visitors and has; performances, tents, music, shopping, sports, boats, cyclists, dancers, parades, talks – you name it and it is probably taking place across the site. Evaluating something like this requires a lot of energy and luckily The Evaluator has been honing its festival approach for many years now. We don’t often get to work next to celebrities like Bluey and Supertato though! Not forgetting the Gladiators and Mr Motivator!

You can read more about the We Invented The Weekend festival here.

four colourful textiles with the letters W I T W in a grid

We worked with an amazing team of volunteers and two giant dice to collect feedback at the weekend festival. We are very proud to say that across two sites, with a really small evaluation team, we were able to collect almost 500 completed feedback surveys during the weekend itself. We also created a brand-new game which worked really well, involving a giant dice and questions which aligned to the dice roll. This means that we have really good results across a wide cross section of people, great sample sizes and very good data.

 

photogragh of Salford Quays which shows tall buildings and blue skies next to water with people walking

Whilst there, we also carried out extensive footfall counting, to allow us to create a robust estimate of footfall over the whole weekend.

Our surveys are still live for this project and we are looking forward to collecting and analysing all the data over the summer months.

You might spot us at the next couple of festivals we are working on; these are ‘The Festival of Culture’ in Nelson in June and also ‘The National Festival of Making’ in July.

You can see more about The Festival of Culture here, and you can see more about The National Festival of Making here.

We actually evaluate the number of festivals and are happy to come along and support data collection if needed. Our festival analysis includes audience profiles, quality measures, practical experience and ways to improve, alongside footfall and economic impact. We would be happy to have a chat if you have a festival, that you would like to know more about, evaluating. You can call us on 01756 532 538 or email: info@theevaluator.co.uk

What are the Indices of Multiple Deprivation?

What are the Indices of Multiple Deprivation?

If you have ever read one of our reports, you’ll likely see reference to something called the Indices of Multiple Deprivation, or ‘IMD’.

We use the IMD as a powerful tool to analyse and understand the barriers and socio-economic status of audiences, participants, and volunteers. We use postcodes to measure the IMD status, and this means that people find it easy to answer. This does mean that we are looking at the household status too.
The results give our clients an idea of which people they are attracting to their projects and what barriers they may be facing in life.

Statistically, people who live within areas of greater deprivation have more barriers in their lives and are less likely to engage with the arts and nature and are more likely to have greater issues with their wellbeing. That’s one of the reasons funders do like to understand if projects and organisations are reaching people from the most deprived areas.
How are the indices of multiple deprivation worked out?
The IMD is worked out using a range of factors which are assessed across the UK and given a score. The factors are listed below:

  • Income deprivation: how many people in the area are experiencing low-income levels.
  • Employment deprivation: what is the rate of unemployment in the area?
  • Education deprivation: what are the average qualifications within the area, and how easy is it to access quality education?
  • Health deprivation and disability: what is the average life expectancy and disability prevalence in the area?
  • Crime: how much crime occurs within the area
  • Housing deprivation: how affordable is housing within the area, and are there other barriers to people accessing housing
  • Living environment: what is the air quality like in the area, and are there green spaces?

These scores are compiled to give a ranking, which are then categorised into ten deciles. The entirety of England falls equally into one of ten deciles, with decile 1 indicating the most deprivation, and area 10 having the least deprivation.
If you were targeting deprived areas, you might want to reach 50% of people living in IMD1 and 2 for example. Or if you wanted a perfect section of the UK represented in your project, you might want to see 10% of your audience from each decile.

We think the IMD is a useful tool and aim to use it in every evaluation where relevant.