Too many times people’s agendas get in the way of measurement.  So often the social difference can be simple to identify – people healthy, people out of poverty, people away from harm, people in employment – the difficulty is the activities that make that difference – what will trigger change?  This is made more difficult when the context/environment can change rapidly without notice. This means flexibility to try different approaches and to learn is vital.  I agree that it needs to be a redefining of measurement – so that measurement is seen as an integral part of the delivery process – where learning what is working and what is not is considered to be a success in itself, and recognising that quick adjustments (as well as longer term adjustments) to an activity that is not working is also a success.  We have to not just look for the feel good story but also the story of what can be been done better/differently.In practice this means deliverers and beneficiaries are the innovators, supported by others that can use their knowledge to help the innovative process and record it.  In practice this also means funders of projects need to encourage a depth of inquiry as an integral part of delivery – a key cost to the project; and a recognition that the impact of the activities might not be what was hoped for and that can be ok.  It also means project deliverers prioritise immediate, short term and long term learning with the people they are supporting – stopping the ‘doing’ to ask what is happening and what could be tried to be more effective – which can be challenging when the day-to-day busi-ness takes over.

Tatyana, Johannes, a systemic view in the way you describe would seem an absolute ‘must’ in evaluation and innovation.  The only thing that I would add is some caution to the approach in practice.  It can be very frustrating when this approach is taken without an acknowledgement of any bias of the assessor and the participants involved.  And even more frustrating is when this approach includes a number of assumptions entwined in the judgements made. The result is a skewed systemic picture, which can then lead to some ill-informed decision-making – as too often seen in political policy making after extensive consultations.  A critical eye on the information gained, the relevance of knowledge shared and what is considered good practice can make a significant impact on the value of the evaluation and innovative design.  

David, I have enjoyed reading your blog.  My thoughts on this is that often intuition can be shorthand for a large collection of qualitative measures that someone uses, often in a very short period of time, based on their experience.  I have found that people can be either reluctant or lacking in practice of isolating the multitude of factors.  Reluctant because they could be qualitative measures that others might disagree with or consider ‘fluffy’; lacking in practice because their expertise has not often been used to teach or explain to others.  Or both!  Breaking down more complex expertise is key to knowledge transfer, succession planning and understanding true indicators of social impact, so it is vital that people are encouraged to share their thoughts on why they may have ‘intuitively’ come to a decision.  

Sally, I really enjoyed reading about how you’ve ensured a
commitment to active learning within the evaluation process.  There is always an energy that comes
through when people talk about their work in this way and I will now be sure to
keep up with your progress on the Ripple Effect.   In terms of the types of evaluation I would argue that
there ought not to be such a distinct separation between evaluation for
validation and active learning, and rather that they are inextricably linked. 

I have worked with a number of non-profit/charitable
organisations and social enterprises (in the UK) who are carrying out fantastic
work within local communities, constantly moving forward and creatively building
on their work.   However when
it comes to their formal evaluation e.g. end of year 1, year 5 etc, they leave the
majority of these progressions out and their evaluation becomes a largely lifeless, often tenuous in meaning, set of numbers and case bites. 

Evaluation for validation does not need to be carried out in
this way – it can be an accumulation of active learning over a time period/s –
a summary of the critical elements learnt (both the challenging and rewarding
ones – which will inevitably include some quantitative data but critically a
depth of qualitative data).  This
requires however a different expectation of what validates a project – is it a
depth of knowledge and meaningfulness or an acceptance of easy to calculate
numbers and sound bites. 

A key to the collation of active learning as an evaluation
for validation is recording.  People
are often less good at recording their insight as it can seriously stifle the
creative process.  However by
recording a summary of key lessons learnt and what this has meant to the
projects development (in a relevant format to the work being carried out to
reduce the stifling) will take less time than doing a separate, from scratch
(or near enough), validation evaluation later – which unfortunately most
projects have to do to satisfy funders. 

It requires a redefining of what evaluation information is
meaningful and it places an emphasis on learning in the short, medium and long
term as key to success.  Funders in
particular need to encourage this in their projects if they truly want
innovative work – otherwise a significant amount of resource will be spent on quantitative
and limited qualitative information that is not actually that relevant or
helpful to their project.  It also
requires a flexibility in targets and a willingness to challenge assumptions on
what is success – and consequently what is the most important information for
decision-making.

Too many times people’s agendas get in the way of measurement.  So often the social difference can be simple to identify – people healthy, people out of poverty, people away from harm, people in employment – the difficulty is the activities that make that difference – what will trigger change?  This is made more difficult when the context/environment can change rapidly without notice. This means flexibility to try different approaches and to learn is vital.  I agree that it needs to be a redefining of measurement – so that measurement is seen as an integral part of the delivery process – where learning what is working and what is not is considered to be a success in itself, and recognising that quick adjustments (as well as longer term adjustments) to an activity that is not working is also a success.  We have to not just look for the feel good story but also the story of what can be been done better/differently.In practice this means deliverers and beneficiaries are the innovators, supported by others that can use their knowledge to help the innovative process and record it.  In practice this also means funders of projects need to encourage a depth of inquiry as an integral part of delivery – a key cost to the project; and a recognition that the impact of the activities might not be what was hoped for and that can be ok.  It also means project deliverers prioritise immediate, short term and long term learning with the people they are supporting – stopping the ‘doing’ to ask what is happening and what could be tried to be more effective – which can be challenging when the day-to-day busi-ness takes over.

Tatyana, Johannes, a systemic view in the way you describe would seem an absolute ‘must’ in evaluation and innovation.  The only thing that I would add is some caution to the approach in practice.  It can be very frustrating when this approach is taken without an acknowledgement of any bias of the assessor and the participants involved.  And even more frustrating is when this approach includes a number of assumptions entwined in the judgements made. The result is a skewed systemic picture, which can then lead to some ill-informed decision-making – as too often seen in political policy making after extensive consultations.  A critical eye on the information gained, the relevance of knowledge shared and what is considered good practice can make a significant impact on the value of the evaluation and innovative design.  

David, I have enjoyed reading your blog.  My thoughts on this is that often intuition can be shorthand for a large collection of qualitative measures that someone uses, often in a very short period of time, based on their experience.  I have found that people can be either reluctant or lacking in practice of isolating the multitude of factors.  Reluctant because they could be qualitative measures that others might disagree with or consider ‘fluffy’; lacking in practice because their expertise has not often been used to teach or explain to others.  Or both!  Breaking down more complex expertise is key to knowledge transfer, succession planning and understanding true indicators of social impact, so it is vital that people are encouraged to share their thoughts on why they may have ‘intuitively’ come to a decision.  

Sally, I really enjoyed reading about how you’ve ensured a
commitment to active learning within the evaluation process.  There is always an energy that comes
through when people talk about their work in this way and I will now be sure to
keep up with your progress on the Ripple Effect.   In terms of the types of evaluation I would argue that
there ought not to be such a distinct separation between evaluation for
validation and active learning, and rather that they are inextricably linked. 

I have worked with a number of non-profit/charitable
organisations and social enterprises (in the UK) who are carrying out fantastic
work within local communities, constantly moving forward and creatively building
on their work.   However when
it comes to their formal evaluation e.g. end of year 1, year 5 etc, they leave the
majority of these progressions out and their evaluation becomes a largely lifeless, often tenuous in meaning, set of numbers and case bites. 

Evaluation for validation does not need to be carried out in
this way – it can be an accumulation of active learning over a time period/s –
a summary of the critical elements learnt (both the challenging and rewarding
ones – which will inevitably include some quantitative data but critically a
depth of qualitative data).  This
requires however a different expectation of what validates a project – is it a
depth of knowledge and meaningfulness or an acceptance of easy to calculate
numbers and sound bites. 

A key to the collation of active learning as an evaluation
for validation is recording.  People
are often less good at recording their insight as it can seriously stifle the
creative process.  However by
recording a summary of key lessons learnt and what this has meant to the
projects development (in a relevant format to the work being carried out to
reduce the stifling) will take less time than doing a separate, from scratch
(or near enough), validation evaluation later – which unfortunately most
projects have to do to satisfy funders. 

It requires a redefining of what evaluation information is
meaningful and it places an emphasis on learning in the short, medium and long
term as key to success.  Funders in
particular need to encourage this in their projects if they truly want
innovative work – otherwise a significant amount of resource will be spent on quantitative
and limited qualitative information that is not actually that relevant or
helpful to their project.  It also
requires a flexibility in targets and a willingness to challenge assumptions on
what is success – and consequently what is the most important information for
decision-making.