Visualising data

We've been thinking for a while about how Lamplight might be able to help you visualise and reflect on the data it produces in reports.  Tables of figures are all very well, and charts can help, but can sometimes be difficult to tell the whole picture.  Infographics (this is one of my favourites, at http://www.informationisbeautiful.net) can help to tell a fuller story.

We're not claiming the beauty of some of the best infographics, and this is just a first effort at helping to make sense of the data that's there (a 'straightforward' Lamplight outcomes report generates up to 18 different statistics for each outcome measure).  Perhaps some of the 'beauty' of this is that Lamplight generates this in one click from your standard reports...

In the first part, the background bar is scaled to the minimum and maximum possible values for the measure, and the arrow shows the average start and average end scores for all our service users (of course this is fictional data).  The second chart shows some distribution: the width of the arrows is proportional to the number of people that saw positive or negative (or no) change; the height is proportional to the amount of change they saw (on average).

We'd be really interested to hear feedback on whether this helps, and what other measures and graphics might also be useful.

Open Data

Sarah's written a piece on open data in the voluntary sector, which has just come out in Lasa's Computanews.

"Open data can enhance accountability, transparency and learning for charities and voluntary organisations. At the same time it presents a series of challenges to the sector. Technical expertise, time and fear are all factors but without the data standards to back it up the open data movement threatens to keep its insights closed."

Read the rest here...

Outcomes Monitoring as Therapy?

Outcomes and impact mearsurement have become hot topics and for good reason: if you want funding you have to show me why you deserve it. What do you do? What do you achieve? What makes your organisation a good (social) investment? Funders and donors offer support, guidelines, frameworks and systems to help organisations to properly evaluate and monitor their work and best show off their achievements and thus achieve more funding (See the recent NPC blogpost Caring and Sharing).

But there is a danger that with this approach, especially among the smaller charities, outcomes and impact monitoring becomes something external, something imposed from the outside. There is a fear of looking too closely as inevitably the bad will be revealed with the good - I heard this view last week at a meeting to oppose cuts to local services. After listing all the amazing things that the organisation had achieved the chair of the meeting expressed the view that monitoring outcomes was a trick imposed by the government to legitimise cuts. Evaluation and monitoring wasn't something they needed to do, it was something someone else wanted them to do.

So why Outcomes Monitoring as therapy? Well, therapy helps you identify the good as well as the bad. To see what's working and to see what needs to change. And you have therapy for yourself, not because your employer or your spouse or your father tells you to. But the effects trickle over into everything: more confidence; better working practices; stronger relationships.

So while organisations will still need to convince their funders that they are value-for-money they can do that because they know who they are, what they do and how and why they do it well. The motivation for outcomes monitoring must come from within the organisation, only then can funders and donors see an organisation at its best.

Impact impact everywhere

Maybe it's just what I tend to read, but it feels like there's been more and more about voluntary sector impact around the place recently.  Last Thursday the Guardian ran an overview piece looking briefly at how any why impact measurement is important.

Of course New Philanthropy Capital have been working on this for ages - and they blogged earlier this month about work they're starting on standardising impact (/outcome) measures.  This is a good thing, in our view, but doesn't need inventing from scratch, and crucially needs to be 'free' - measures should be free for any and all to use.  The value of this sort of work would be much increased if data formats could also be agreed to easily allow #opendata learning and benchmarking.

Often the angle that these discussions come from is funding - like this one from NCVO discussing new forms of funding (e.g. social impact bonds) and the link with outcomes, and a similar brief on new forms of funding from Third Sector Foresight.  This is clearly a valid and important perspective (and again from the funders mouth itself - this time the Chief Exec of the Big Lottery Fund) - but it seems rarely to be said that understanding your impact is a good thing for staff, trustees, and probably beneficiaries of a charity too.  We'd like to hear a bit more about these internal benefits.

So my sense is that impact/outcome measures are going to become more and more important.  And in my view, NPC are correct that standards are really important in this discussion.

Firstly, it can be really hard to measure some of this stuff.  At least, it's hard to do well.  Using measures and approaches that have been developed and tested rigorously makes sense.

Secondly, all these measures are of little use at all if they are not communicated, internally and externally.  If you're developing your own impact measures, you've got to explain what and how and why you're measuring, as well as the results.  Which means you'll lose people.  If a set of standards emerges that become more generally understood by trustees, staff, funders and others, it becomes much easier to talk about your impact. And if those standards are trusted (which they'll really need to be) questions over methodology etc. should all-but disappear.

Thirdly, they allow comparison.  This is clearly more controversial: charity league tables anyone?  Martin Brookes at New Philanthropy Capital has spoken and written roughly along these lines - though ranking causes, not individual charities.  But one of the things NPC does is evaluate (ie compare) charities for funders.

There are two kinds of comparison: competitive (we're better than you) and collaborative (what can we learn from you?).  The second shouldn't be too scary.  The first happens already - whenever a funder makes a choice to fund one organisation and not another - it's just that the comparison and process is fairly opaque.  So perhaps it shouldn't be something to be scared of.

This kind of benchmarking would help organisations identify, communicate and celebrate their strengths, and know who to learn from to address weaknesses (or what to stop doing).  It will be rather controversial and perhaps painful.  But will it be a good thing?


[Tuesday 22nd update] The Guardian had a Q&A on this today: http://www.guardian.co.uk/voluntary-sector-network/2011/mar/18/q-a-measuring-impact?commentpage=all#end-of-comments and http://twitter.com/#!/search?q=%23volsecqa (though not much action on twitter).

Charities Evaluation Service - impact frameworks

Charities Evaluation Service have just published an overview of outcome frameworks available, as a free download.  We like it - it's useful background reading if you're thinking about how to measure impact.  The problem with this sort of thing - and I know CES know it - is keeping it up to date - New Philanthropy Capital have been doing some work on measuring impact on well-being of young people, and there's work going on for infrastructure organisations there and at NCVO.

We hope NPC are going to make their 'well-being' work public and freely available - the background on the methodology looks to be some of the most rigorous we've seen.  And I know that some of our customers working with young people could use it.

What to do with Impact

Just read this from Richard Piper on one of the NVCO blog:

Second, there's a critical distinction between evaluation and impact. You may have noticed that the title of this blog refers to impact, but what I've actually been talking about is evaluation. If impact and evaluation were broadly the same thing, that wouldn't be a problem. However they are very different, as I've argued elsewhere (http://bit.ly/ImpactLeadership). 'Impact' can of course be evaluated, but there are other things you can do with it that are at least as important: plan for it, deliver it, improve it, communicate it, and place it at the heart of your organisational culture.  Too many people are put off thinking about impact because they mistakenly think it’s primarily about evaluation, monitoring and measurement.

In particular, properly planning what impacts we want our organisation to create is absolutely vital if we are to fulfil our potential and really make a difference. For organisations that are new to impact thinking, one of the most common challenges is moving from asking "right, given our products, services and other outputs, what impact do we create?" to asking "right, let's decide what changes we want to make in the world, and then work out what services and outputs would best deliver that impact". Impact thinking can feel very unfamiliar and unsafe to some people who've perhaps spent years or even decades stuck in outputs thinking.  But this thinking is, in my experience, the most fundamental characteristic of genuinely successful voluntary organisations and social enterprises.

We're working with NCVO (and Richard) at the moment on their Value in Infrastructure programme - a toolkit to help infrastructure organisations plan, measure and communicate their impact.

It strikes me that impact is going to be more and more important - whatever you think of the Big Society, it does look like there might be opportunities for the third sector, eventually - but if there is, commissioning by impact may well become the norm, perhaps along the lines of the social impact bond launched a little while back.

Interesting for us, as a business too (we're revising our business plan at the mo) - the impact we want to have is 'better charities' - so what do we add to Lamplight, our training, and other services to deliver that?  I think it's always in our heads, but perhaps we should put it up front a bit more.

('Better charities' is a shorthand - not trying to sound arrogant!)